0:00
User-centered performance refers to customer obsession or user-centered practice that is symbolic rather than focused on learning.
Search this video
0:06
It's hugely common, I would argue.
0:06
It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision.
0:17
If your listeners are like, "I don't do that."
0:17
I'm like, "Think about it for a second.
0:17
This is extremely common."
0:23
Every time a PM comes to a researcher at the end of a product process and says, "Can you just run a quick user study just to validate our assumptions," that's user-centered performance. It's too late to matter. We got to ship it.
0:33
What they want is to check the box.
0:33
One of my big mantras was, "We don't validate, we falsify.
0:38
We are looking to be wrong."
0:38
Many PMs, many designers are not in that place.
0:45
They do not want to be wrong.
0:45
They're looking to validate, and that's user-centered performance.
0:50
Today my guest is Judd Anton.
0:50
Judd helped build the user research practice at Facebook.
0:57
He was a longtime head of research at Airbnb, and his direct reports have gone on to lead research teams at Figma, Notion, Slack, Robinhood, Duolingo, Fair, and other amazing companies.
1:08
These days, Judd spends his time consulting, helping companies with organizational challenges, product strategy, design, research, hiring, onboarding, and crisis management.
1:19
In our conversation, we unpack a conclusion that Jud has come to recently about how the user research field is going through a reckoning and what needs to change both within the user research field and how companies leverage user research going forward.
1:37
Judd shares what the user research field has gotten wrong over the last decade, how PMs and designers rely on user research too often, and to answer the wrong questions, where user research will continue to provide significant value, and how to best leverage your researchers, why it's important for researchers to think about the business goals more versus just what the users
1:57
need, what to look for when you're hiring a user researcher, how PMs can be better partners to researchers, and also a phenomenon that I love that Judd describes and often witnesses, that he calls user-centered performance, where everyone acts like they care about the user, but they're just doing it for show and already know what they want to do. This episode has a lot of
2:11
This episode has a lot of spicy takes and will probably upset some people, but Judd is sharing some real talk, here, that I think we all need to hear.
2:21
With that, I bring you Judd Antin after a short word from our sponsors.
2:27
This time of year is prime for career reflection and setting goals for professional growth.
2:27
I always like to spend this time reflecting on what I accomplished the previous year, what I hope to accomplish the next year, and whether this is the year I look for a new opportunity.
2:36
That's where today's sponsor Teal comes in.
2:41
Teal provides you with the tools to run an amazing job search with an AI powered resume builder, job tracker, cover letter generator, and Chrome extension that integrates with over 40 job boards, Teal is the all-in-one platform you need to run a more streamlined and efficient job search and stand out in this competitive market.
2:56
There's a reason nearly one million people have trusted Teal to run their job search.
3:00
If you're thinking of making a change in the new year, leverage Teal to grow your career on your own terms.
3:05
Get started for free at tealhq. com/lenny. That's tealhq. com/lenny.
3:11
This episode is brought to you by Vanta, helping you streamline your security compliance to accelerate your growth.
3:20
Thousands of fast-growing companies like Gusto, KOM, Quora, and Modern Treasury trust Vanta to help build scale, manage, and demonstrate their security and compliance programs, and get ready for audits in weeks, not months.
3:36
By offering the most in-demand security and privacy frameworks such as SOC 2, ISO 27001, GDPR, HIPAA, and many more, Vanta helps companies obtain the reports they need to accelerate growth, build efficient compliance processes, mitigate risks to their businesses, and build trust with external stakeholders.
3:52
Over 5,000 fast-growing companies use Vanta to automate up to 90% of the work involved with SOC 2 and these other frameworks.
3:58
For a limited time Lenny's podcast listeners get $1,000 off Vanta. Go to vanta.
4:03
com/lenny, that's V-A-N-T-A.
4:03
com/lenny to learn more and to claim your discounts. Get started today.
4:12
Judd, thank you so much for being here.
4:20
Welcome to the podcast, Lenny, thanks for having me. It's my pleasure.
4:23
So we actually worked together at Airbnb for many years.
4:23
And as I was preparing for this, I realized how many of the people that you managed went on to do amazing things.
4:34
So I'm just going to read a list of people that worked for you and what they do now.
4:38
We had Matt Gallivan, who now leads research at Slack.
4:38
We have Janna Bray, who leads research at Notion, Celeste Ridlen, who leads research at Robinhood.
4:43
Rebecca Grey, who leads research at Fair, Hannah Pileggi, who I think was leading research at Duolingo, Louise Beryl, who leads research at Figma, and then Noam, who was leading research at Wealthfront.
4:57
I think he moved on to something else.
5:03
What a fricking crazy alumni community and group from this one team that you hired and incubated.
5:08
No, I've never looked at that list, but I'll tell you, I have been so privileged to work with all these amazing humans.
5:12
I can't take credit for it.
5:16
They're just outstanding people.
5:16
And I'm glad the diaspora is out there, because these people, rock stars. Okay.
5:20
The main reason that I wanted to do a podcast episode with you is that you wrote this piece that was titled The User Research Reckoning is Here, which I understand caused quite a stir in the research community and I think adjacent communities.
5:36
And let me just read one of your takeaways at the top of your post to give people a sense of what it was about.
5:42
You wrote, "The user research discipline over the last 15 years is dying. The reckoning is here.
5:46
The discipline can still survive and thrive, but we'd better adapt, and quick."
5:53
Before we get into the meat of the piece, could you share a bit about just the reaction to this piece and maybe if it was a surprise and what you expected would happen when you put this out?
6:03
Yeah, I was definitely surprised.
6:03
I wrote it because I wanted to start a conversation about something I was thinking about.
6:10
I didn't really know who would read it.
6:13
And in the end it turned out a lot of people read it.
6:18
I learned that using the word reckoning may have been a mistake because it inspires a lot of drama in a conversation that I wanted to be really productive and positive.
6:24
Overall, I would say, though, that the response was very positive.
6:29
It seemed to resonate with a lot of people who reached out to me.
6:35
I spent a lot of time talking to teams, to designers, to researchers, but there were also a ton of critiques.
6:40
I would say some of it was like people thought I was throwing research or researchers under the bus, like, "It's researchers' fault. We're doing it wrong."
6:49
Which I don't believe at all.
6:49
And that I wasn't taking responsibility as a research leader or a design leader myself.
6:54
And the most interesting one I would say was the anti-capitalist crew, because one of my points that we'll talk about is that I think researchers need to be more profit focused.
7:06
And there are a lot of people out there who, I think they think that's not cool or not research's job, and I'm like, "Well, what are we doing then, if we're not helping businesses succeed?"
7:17
But that was the most surprising critique, for sure.
7:23
I've worked with some of those people who are just like, "Why are we growing?
7:25
Why do we focus so much on growth?
7:25
Why do we need to grow this business?" Yeah. So I get that.
7:29
Maybe it's the wrong industry for them. Yeah. I'm not a fan of that. Okay.
7:29
Let's actually dig into the meat of your message, and the big takeaway, and the conclusion of what you're finding is happening in user research.
7:39
And I know a lot of this comes from a lot of user researchers have been laid off at a lot of companies.
7:44
It was one of the hardest hit teams.
7:44
And so I think a lot of this comes from that.
7:49
So yeah, so let's just start big and then see where it goes.
7:52
So yeah, everybody who's paying attention has noticed that there have been a bunch of layoffs.
7:57
And I think back in the summer I was thinking, "Listen, this seems to be hitting UX and UX research particularly hard.
8:02
Is there something going on?
8:02
Is there a bigger picture?"
8:07
The reason I use the word reckoning is because to me that's like, "Hey, a moment to take stock."
8:12
And triggered by the fact that a lot of wonderful humans may have lost their jobs, and many more are afraid of losing their jobs.
8:18
And so if it's a sign, the fact that research has been hit so hard, it's a sign of what?
8:23
And so the thesis of my article is really, it's a sign that maybe the system is a little more broken than we think and that research is not driving the value or impact that it should or could.
8:33
And that's for a bunch of reasons, I think.
8:39
Some of it is stuff that research can do better, and a lot of it is how research is integrated and positioned in companies.
8:43
And at the root of all that, I think, is that we're just doing too much of what I would consider the wrong type of research.
8:48
And what I mean by the wrong type of research is I have this framework, and it's in the article, macro, middle range, and micro research, at least three ways to talk about it.
9:00
And it's pretty simple, the intuition of what those are.
9:05
So macro research is big picture, strategic, business focused, forward-looking innovation, look at the market, look at competitors, long-term research to understand where the product should go next, stuff like that.
9:20
And then you have micro research, which a lot of really technical usability falls into this, all the beautiful stuff that researchers do to enable a really high quality, excellent, pixel perfect thing to go out the door, laser-focused research to understand AB test results, stuff like that.
9:35
And then you have this middle range, which is this blobular place where the research questions are middle altitude and a lot of the core, let's say user understanding questions fall here.
9:48
And a lot of what research is doing is research in that space.
9:48
It's, "Let's take a group of people and ask some questions about how they think, feel, behave, how they're using a product or not using a product."
10:00
And it's just this devastating mix of really interesting to many, including researchers, and not impactful enough for the business. That's the core thesis.
10:13
Researchers do it because it's interesting, but honestly, and a thing we should talk about, Lenny, is researchers also do it because it's the kind of work we most often get asked to do. Yeah.
10:22
That's exactly what I was thinking.
10:22
As a PM, that's what I want to get answers to, is like, "How should we think about this one product?" And I totally get this. Yeah.
10:31
The questions turn out to be really interesting and there are many cases at many companies where it's super impactful.
10:36
But the problem with those types of questions is they tend to, they trigger all the worst stuff that researchers experience.
10:41
So they yield results which are interesting, but sometimes hard to operationalize.
10:48
They trigger the post hoc bias, really, really, really well where a lot of people can say confidently like, "Oh, that was obvious. We knew that already."
10:59
And they fulfill this need for us to feel and be customer obsessed, user-centered, without changing anything.
11:05
So doing too much of that research to me is a symptom of a broken system, and where companies are really different from each other.
11:17
I heard from so many after this article and they're like, "Well, my company and my industry is like this or not like this."
11:21
But in tech, we spent the last many years hiring, hiring, hiring researchers, but maybe, I'm sure most of your listeners are familiar with the idea of a ZIRP.
11:32
Maybe it was a zero interest base phenomenon, where it was okay when the money was easy, to hire researchers, even though we were not setting them up properly.
11:38
We're were going to set them up to fail.
11:44
We set them up as a service function.
11:44
We didn't know what research was for.
11:49
We didn't know how to really drive impact with it.
11:49
And that's where the reckoning comes from.
11:53
It's like that era is over.
11:53
Research, I think, is more crucial than ever.
11:53
Good, great researchers are more impactful than ever.
12:00
But it's in a new space.
12:00
We're in a new space now.
12:05
I want to make sure people understand this framework.
12:05
And specifically, how would you best describe the difference between this middle range research and macro research?
12:16
Middle range research is usually focused on a more specific set of research questions or a constituency.
12:21
So if macro is like, "Let's understand the overall competitive landscape.
12:28
Let's do a concept car type project where we really look ahead.
12:28
Let's get involved with strategic planning," which is a wonderful thing for researchers to do, do TAM studies, other things like that, that stuff lives in the macro space.
12:42
The middle range space is like, what's a good example?
12:42
"We want to know how Airbnb hosts feel about their payment options."
12:49
That's a really interesting, reasonable question.
12:49
And we can go out and do research on that, but it's not that specific.
12:55
It's not really targeted at a business problem yet. It could be.
13:00
Maybe that's a result of the research, but it yields these middle range insights in which we've learned things like, "Well, hosts want flexibility about their payment options." I'm making this up.
13:13
And that's a good example where it's like, "It's not that that's not an interesting set of questions, it's just not quite pointed enough."
13:18
And it's not framed in the language of the funnel, or the business strategy, or the OKRs.
13:25
It's not quite enough aligned enough to that.
13:32
It's too blobular in that middle level and it ends up not driving impact.
13:37
I think it also leads to a lot of the things as you described, people don't like about research. It delays everything.
13:42
You have to wait for the research to be done to have an answer, to make a clear decision.
13:47
It also creates this issue that people complain about, that PMs and product teams don't want to just make a decision on their own.
13:54
They're like, "I will get this additional data point and make sure research tells us this is the right answer instead of just trusting there."
13:57
God, I guess maybe along those lines, this might be going off a little track, but what's your advice there for, say, product managers or PMs or product teams to not necessarily rely on research for that middle research?
14:11
I think the reason why so many PMs ask for those middle range questions is because they haven't really gotten deep with their researcher in a way which can leverage it for maximum impact.
14:22
So if the question is like, "Hey, Judd, you just pointed out a bunch of problems, can you be more solutions oriented?"
14:27
Well, the solution is simple but not easy to me.
14:32
It's that we need to restructure the way we make products in a way which integrates research much more fully.
14:40
It looks like consistent relationships in which researchers, and the work, and the insights they provide are a part of the process from beginning to end.
14:51
And I think, Lenny, you as a PM, that's how you worked.
14:51
I remember you, I know who you worked with.
14:57
You worked with great researchers.
14:57
But honestly, most product processes are not that way.
15:02
And so that's when research is a service function.
15:02
It gets called in right at the end.
15:08
It's reactive in the sense that a researcher in the room listening and participating in the conversation could have a ton of impact on framing exactly the right question that will drive maximum business impact, maximum product improvement at that moment, and then go do it quick, and get back, and we're onto the next.
15:23
But they weren't there, the relationship wasn't there.
15:23
They're not engaged in the project from the beginning.
15:28
And that's the number one root of the problem.
15:33
As long as research is a service discipline, I think we're going to be stuck in this spot.
15:37
When people might be hearing this, on the one hand, it's research has been not as helpful to teams as they thought, and researchers have been spending time on the wrong thing.
15:47
On the other hand, your advice is integrate research from the beginning, make them more involved throughout.
15:50
And I think that might confuse people.
15:50
How should people think about, like, "Research is actually more important?
15:56
You should integrate them more deeply."
15:59
There's a vicious cycle that's been happening, is from where I sit, and this is what I hear from many, many researchers and research leaders, which is a lot of companies hired a lot of researchers with great intentions, didn't quite know how to integrate them.
16:09
And UX research is a newer discipline, so maybe that's not surprising.
16:16
We're still learning how to use it. "Cool, let's evolve."
16:20
But a lot of companies hired these people, but they hired them into kind of like a service discipline, very reactive, not in the room, not integrated in the way I said.
16:29
And so they had less input on the questions to ask, or they're included, but only at the end.
16:35
And then they're unable to build those direct relationships, to be there in the room to actually drive the questions and insert insights.
16:39
Because a good researcher is like the repository of insights you need for growth, but they're not there.
16:44
They don't participate in the decision.
16:44
So they end up doing research.
16:49
They have jobs to do, so they do research that is too reactive, it doesn't matter, and then it's less impactful.
16:54
Executives conclude that therefore researchers are not as impactful and then they get sidelined or laid off and the cycle continues.
17:07
So I think the short circuit is the constant engagement.
17:07
If you take a great researcher and you insert them consistently in a product process, I feel confident that researcher will drive a product improvement, metrics impact, growth, all the things that you want to see as a PM and a product leader.
17:25
It's just that's the exception, not the norm these days.
17:31
This may be a hard question to answer, but when people hear, "If you have a great researcher, here's how you approach it."
17:35
What are signals that your researchers is great versus not great?
17:38
What are some things people could look for to tell them, like, "Oh, maybe I have the wrong researcher on my team."
17:42
The best researchers I think are first of all, multi method.
17:47
The first iteration of user research was primarily a qualitative discipline.
17:47
But a strong opinion that I have is that is largely one of those models that needs to evolve.
17:53
It's not that qualitative user research is no longer important.
17:59
It's that the best researchers have five tools.
18:04
I think they have five tools.
18:04
And those five tools are number one, what we would call formative or generative user experience research.
18:10
So looking ahead, innovation focused, really open-ended, maybe more ethnographic, "Let's go out into the field and talk to host and guests on Airbnb.
18:20
Let's see people using our product in the field," stuff like that. So that's formative.
18:25
The second type is evaluative, so more like usability testing.
18:31
The third tool is a basic rigorous survey design.
18:31
It's the best scaled way to get responses from communities small and large.
18:38
You can get a lot out of really well crafted surveys.
18:43
But to do that, you have to have the fourth tool, which is applied statistics, the best research, know a little bit of stats.
18:49
You can't interact in a world of AB testing without knowing basic statistics.
18:52
And then in the old version of this, the fifth tool was SQL, because I think good researchers need to be able to run their own queries.
18:58
These days, so much of that is dashboarded, that the fifth tool may now be prompt engineering, which is a thing we could talk about, but I think maybe that's the fifth tool is somewhere, is it technical skills that fall in between querying your own data, understanding it very well in companies that are awash with data and then interacting with generative AI. Amazing. That's such a cool list.
19:23
Okay, so just to playback, formative, generative, innovative skills to think bigger and come up with new ideas, usability. Yep. Yeah, usability. How did you describe it?
19:35
I have a different word, here. Evaluate? Evaluative? Evaluative, right. Okay.
19:41
So we're evaluating products and doing more.
19:43
Really that's the micro level of research.
19:43
Survey design, being really rigorous about it, applied statistics, and then SQL/dashboard/prompt engineering. Right.
19:53
Maybe just one last question along this thread, also a big question, but any advice for how to evaluate these skills/interview for them?
20:01
I know this is its own deep topic, but any advice for someone trying to find this person?
20:05
I've interviewed hundreds or thousands of researchers, and the way I usually approach that is you want a researcher who's got a Swiss army knife, because if all you have is a hammer, then everything looks like a nail.
20:15
And so if you give in the context of an interview, let's say, a researcher, a pretty juicy, open-ended research question, and you want to see how they handle it, and a good answer is usually multi-method.
20:27
We're not going to handle it in any one way.
20:27
We're going to say, "Well, here's a couple of ways we could deal with this.
20:32
Here's how we could do this in a day, or a week, or a month."
20:35
We usually don't have a month, but sometimes big research projects go on for that long.
20:40
"And here are the different sets of methods that we can use." So see where they go.
20:44
It's actually pretty simple.
20:44
Most researchers are deeper in one than the other, and sometimes you can make up for those five tools with the team.
20:48
So you have experts who are t-shaped, but maybe deeper in one or several of those ways.
20:53
But when I built a team at Meta and at Airbnb, that was my goal, is individually as researchers build up those tools and then as a team build deep expertise that would fill all the gaps.
21:10
Coming back to the main premise of your post, one of your big takeaways is, "Researchers need to be much more business oriented, thinking about what helps the business versus the user."
21:15
Which I think to a lot of researchers will feel really weird.
21:20
Can you just talk about your takeaways there?
21:25
So much of user experience practice, not just research, but design too, is focused on empathy and very user-centered. This is beautiful.
21:33
I'm not saying that we should abandon that.
21:33
I think what I'm saying is there's an overlapping event, where you have the user and profit or the business.
21:44
And what researchers need to do is be way more explicit about finding that overlap.
21:44
So one thing, when researchers ask for advice, they're like, "Well, what should I do to be more business or profit focused?"
21:57
I say something like, "Did you read the last quarterly report, If it's a public company?
22:04
Did you listen to the shareholder call?"
22:04
And they're probably like, "No, it's full of a bunch of language I didn't quite get." And I'm like, "Yep." So there you go.
22:10
That's the language you need to learn.
22:15
Scour your Google Drive folder, your internal folder and look for all of the documents that are about this quarter, or this halves, or next half strategy. What are the OKRs?
22:24
Understand the metrics and the conversion funnel, know it back and forward, because then what you're doing is you're proposing, if you're in the active conversation, you're saying, 'Cool, I hear you asking that research question.
22:38
I've identified this is exactly the spot in the funnel where I think we need to do work.
22:44
There's an opportunity here.
22:44
Or that competitor is eating our lunch with this group of users.
22:44
I know that because I read the competitive report and I understand it deeply.'"
22:50
So those are skills that some researchers have and a lot are building these days, but historically, last 15 years, it hasn't been a thing we've been as focused on, and I think that's an evolution that needs to happen.
23:06
I think a lot of PMs listening to this are going to be like, "Hallelujah."
23:06
This is exactly what I've been trying to convince people of.
23:11
It's what I've been trying to convince my researchers of, and design often falls into this.
23:14
But Lenny, the opposite is true, too, because you got to take the average PM who lives in that land, all day, every day, and what they do is not in the Venn.
23:24
I think those are people who are also performing customer centricity and performing user-centeredness a lot, when they're really not interested.
23:34
And so this is not about researcher. This takes two sides.
23:40
Fixing this broken system takes everyone, researchers, PMs, designers, everyone at a company, but also the way that organization is structured, and integrating itself in a different way.
23:51
Everybody's got to come to the table. Such a good point.
23:55
And you have this actual term that you call user-centered performance, where it's the performance of being user-centered.
24:00
Can you talk about that and then just what advice you'd give to PMs that, hearing this, are like, "Yes, I love everything you're saying," and then not realizing maybe they're too far in that extreme?
24:11
User-centered performance is a term I made up, because it's fun to make up terms.
24:11
And it refers to customer obsession or a user-centered practice that is symbolic rather than focused on learning.
24:27
So it's hugely common, I would argue.
24:27
It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision.
24:34
And if your listeners are like, "I don't do that."
24:40
I'm like, "Think about it for a second."
24:40
Because this is extremely common.
24:40
It shows up in explicit ways and implicit ways.
24:49
So explicitly, I would say every time a PM comes to a researcher at the end of a product process and says, "Can you just run a quick user study just to validate our assumptions?"
25:00
That's user-centered performance. It's too late to matter.
25:05
That PM is not interested in being wrong at all.
25:05
It's too late in the game for that. We got to ship it.
25:13
What they want is to check the box.
25:13
So any check the box style research is a wild example of user-centered performance.
25:18
I would argue every researcher has probably had to do executive listening sessions because a lot of PMs, founders, product people, but designers, too, they want to get close to the customer.
25:31
And so, like, "Can I do some focus groups? I want to be there.
25:36
I want to ask them questions." This is 97% performance.
25:36
It's well-intentioned, but it isn't focused on learning.
25:42
It isn't going to drive better outcomes or more impact.
25:49
And then there's all these implicit ways that people engage in that kind of user performance, too.
25:52
A lot of it comes down to cognitive biases, confirmation bias, ego.
25:52
One of my big mantras was, "We don't validate, we falsify.
26:02
We are looking to be wrong."
26:02
That is the mindset you should use when you're approaching insights and research. "I want to be wrong.
26:08
I want you to do research that shows we were off base in the following ways.
26:13
Tell me exactly how and why in a way that allows me to fix it quickly."
26:18
But many PMs, many designers are not in that place.
26:23
They do not want to be wrong.
26:23
They're looking to validate.
26:23
And that's user-centered performance. Oh, man.
26:28
I think a lot of people are hearing this and feeling exposed. Exposed.
26:33
I feel like you're like this Deep Throat person coming from sharing these things people don't want to talk about at the office. I know.
26:42
There's this quote in your post I'm going to read.
26:42
"Product managers love to ask for middle range research that they can use to justify decisions they're reluctant to make on their own.
26:47
User designers love to ask for middle range research because it fits their model of what proper design process should look like.
26:55
Executives love to ask for middle range because they don't really understand what research is for, and helps them do performative user-centeredness.
26:59
In the end, they will decide based on their own opinions."
27:04
There is an important place for intuition in product development, of course.
27:11
The best designers, researchers, product people develop strong intuition for the product.
27:17
But you got to understand, intuition is where all of those biases lie.
27:23
It's where all your blind spots are.
27:23
And what great insights people do, what great researchers do when you're next to them all the time, is they'll expose you.
27:29
I don't have to be the Deep Throat, because you have somebody who's professional job is ...
27:33
Keeping you honest is probably the wrong way to put it, but as somebody whose capabilities are about expanding your horizons, making it so that your intuition is constantly improving, you don't have to rely on it when your intuition and the evidence sort of collide in a way that either affirms or falsifies the product decision you made.
27:55
Now something really good is happening.
28:02
And the other thing that is inherent in that quote is I, at Airbnb, wore many hats over the years.
28:07
I was head of research two different times.
28:07
I was head of design for guest products.
28:07
And my last job was I was head of the design studio, so UX research, UX design, writing, localization, they all reported up to me.
28:17
So I've seen this from many disciplinary angles in the UX field.
28:17
And researchers aren't the only ones who are guilty of this.
28:24
I would say design has a ton of performance.
28:31
And it comes from the fact that we have figured out user-centered design, this process, or design thinking, which IDEO popularized.
28:36
Like, "That's what we're supposed to do, right?
28:42
Bezos told us that we, as PMs, had to be customer obsessed.
28:42
So that's what we're supposed to do."
28:48
It's a really common and damaging thing when we don't genuinely have that growth learning mindset, and it's easy to sideline researchers.
28:55
We don't need them in that situation. We've got our guts.
29:00
Isn't the gut where a great PM, a great founder needs to have that gut?
29:00
And they do, but they need to be open to the fact that your gut, is limited, and biased, and narrow, and wrong sometimes.
29:12
The two sides of this is trust your gut opinion, "I don't need research, I don't need data.
29:12
I have opinions, and my own experience, and I'm going to use the product, and let's just go with what feels right to me."
29:22
Versus pure data-driven research driven for designers that are maybe listening for product managers.
29:28
Do you have any advice for just where to fall on that spectrum and just how to best leverage research to inform that opinion?
29:32
Yeah, I taught a class at UC Berkeley this semester on leadership, and we talk about that a lot, because great leaders develop intuition.
29:47
It's the pattern matching part of experience, where you develop heuristics which allow you to make good judgments even if you can't quite explain where that judgment came from. That's what the gut is.
29:57
But it's also, like I said, where bias comes from, where all the cognitive biases, there's a list of 151 of them on Wikipedia, I won't name them, but all those thorny things that lead us astray, the behavioral economists and social psychologists study, those live in the gut.
30:14
And so the advice is when you are looking to check your gut, you have to do that thing.
30:14
A lot of your listeners have probably read Thinking Fast and Slow, System 1, System 2. Right?
30:29
I have it here, right under my laptop, actually, holding up my laptop screen.
30:32
That's so appropriate, Lenny.
30:32
So the secret is not that sexy. It's System 2.
30:32
So you engage that slow, methodical process in which you do analytic thinking as a means of checking your gut.
30:41
Slow in the grand scheme of things.
30:48
Slow meaning not a split second decision, not like months of analysis. That's not what I mean.
30:51
The other thing you can do, and there's really great research on this, is you bring in the wisdom of the crowd.
30:55
So the wisdom of the crowd is a phrase a lot of people are familiar with, and it works in a specific situation.
31:01
The wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table.
31:14
Obviously, if everybody has the same sources of information, then it doesn't matter how many people are out there.
31:18
So if you want to check your gut, get a bunch of different guts together, get a bunch of different people in the room who can bring evidence and intuition to bear, and have an open, direct end kind conversation in which we might disagree.
31:33
You know who's great at that?
31:33
Researchers Leading those discussions essentially, and getting a bunch of people's opinions.
31:41
Yeah, this is the structural solution I'm talking about, Lenny, is like, "I never asked for research teams to have their own separate OKRs."
31:46
I said two things, "Number one, what's the teams?
31:54
Shouldn't the PMs, the engineers, the designers and the research, everybody should have the same set of metrics for success because either we're doing it together or we're not."
31:58
And then I said, "My metric for success is when they won't have that meeting without you."
32:03
That's my metric for success.
32:10
If they cannot have that decision making meeting without the researcher there, that means you've developed influence, strong, trusting relationships, you're an active participant in the process, not just somebody who provides input into someone else's process.
32:20
And that is when researchers can have huge impact.
32:26
I think of the PM role in a similar way, even though people won't have these meetings with PMs, because they're often at the center of lot of the stuff, but you want to be a PM that people want on their team.
32:34
There's a lot of teams that are like, "We don't want and PMs, we don't need product managers.
32:39
They just get in the way."
32:43
And I find that that's only the case when the product manager's not great, and not really good at their job, because most great PMs just make everyone's life easier. They do.
32:50
The grease, I- The grease. ... love it.
32:52
You mentioned also, before we started recording, that the biggest challenge for user researchers is in their relationship with their product manager.
32:59
Can you speak to that and what you've seen there?
33:03
I'm wary of overgeneralizing, but I can tell you that from my experience and from what I hear, the product research or product insights relationship is one of the most challenged.
33:10
And I think it comes from the fact that fundamentally, many researchers are just not included in the process that PMs are running.
33:22
And then, actually, I did some asking around before this podcast, and so I thought, "There are some tropes that researchers have about PMs that are worth PMs knowing, just like four or five of them, the things that researchers know PMs say, which drive us nuts because they're not true."
33:47
So the first one is that research just slows us down. Research is too slow. This is bullshit.
33:55
A great research team can do research in a day, a week, or a month.
34:03
It just depends on what you want to get out of it, like, "How much detail do you need?
34:10
How many people do we need to talk to?
34:10
What is the depth or breadth?
34:10
Do we need to go to seven different countries to talk about our constituencies in Latin America?"
34:14
Well, that's not going to happen overnight, but we don't often need that.
34:18
The other way to look at that is that is it slower to get it wrong and fix it than to take a hot second to do the work to get it right the first time? So that's BS.
34:29
Good research doesn't slow us down, it speeds us up.
34:36
And also just along those lines, a big part of your premise is you don't need to do as much research as people are doing, like this middle research that a lot of the time is put into. Yeah.
34:45
Research can go super fast.
34:45
I think especially, so the macro level research, I hope what it is tied to things like annual planning processes.
34:51
We did a thing at Airbnb several years that we called, it was like Insights 2019, Insights 2020.
34:57
They were concept car projects.
34:57
And we spent quite a long time synthesizing the entire year's worth of insights from every place we could get them and then developing with designers and engineers like a concept car for five years in the future.
35:13
So that's a long process.
35:13
But the micro level, there's so much business value to be derived there, so much business value, and it can go so fast, Lenny, it can go so fast.
35:27
You can have results in 48 hours on these things.
35:27
We did a thing at Airbnb.
35:27
There's a famous story which I'll only tell in the abstract, because I don't want to out anything, but we call it the multimillion dollar button.
35:38
And basically we did research which revealed that people weren't going down the purchase funnel because they were afraid.
35:47
The calls to actions on the button was making them afraid that it would initiate a purchase when really it was just taking the next step.
35:58
We changed the text on the button with help from our amazing content design, our UX writing team.
36:02
We basically changed seven characters and made Airbnb millions of dollars, because what we found out was really simple.
36:11
It was just like, "Hey, this button feels scary.
36:16
The CTA on the button feels scary."
36:16
So that's a great example of how micro ...
36:16
And that happened in like 48 hours, we would discover that insight, or overnight, basically.
36:20
And we were like, "Hm, maybe we should test some other CTAs."
36:25
We did the conversion, we added like 1%, which is really, really hard to do.
36:31
So that's a quick example of how that type of quick research can drive a huge amount of business value.
36:36
This episode is brought to you by Ahrefs.
36:36
Many of you already know Ahrefs as one of the top tools for search engine optimization.
36:42
It's used by thousands of SEOs and companies like IBM, Adidas and eBay.
36:46
What you may not know is that there's a free version that was made with small website owners in mind.
36:52
It's called Ahrefs Webmaster Tools.
36:56
It's free and it can help you bring more traffic to your website.
37:02
Ahrefs Webmaster Tools will show you keywords that you rank for and backlinks that you can get.
37:06
It also performs automated site audits to find what issues prevent your website from ranking higher on Google.
37:11
Every detected issue comes with a detailed explanation and advice on how to fix it. Visit ahrefs.
37:16
com/awt, set up a free account, connect your website, and start improving it. That's A-H-R-E-F-S. com/A-W-T.
37:23
So just to make this even clearer, I think this middle research zone is the stuff that does slow people down, I imagine.
37:32
It's like, "What are the challenges hosts have with payments on Airbnb?"
37:37
What you're basically saying is, "Spend your time doing the micro stuff like usability research and then the bigger stuff that's part of overall planning.
37:49
That's part of the planning cycle.
37:49
It's not like every project you're working on, you need to have a whole research project on." Exactly.
37:52
The micro research should be much more common.
37:58
A lot of researchers think that that's scut work, that usability is something junior researchers do. I completely disagree.
38:05
I think we need to get back there as an industry and be like, "When you make a product easier to use, when you discover problems with functionality, business metrics we care about will go up." I've seen it happen.
38:17
But that's not just work for interns and new grads, that's for sure.
38:21
And then the planning process, absolutely.
38:21
If we're integrated from beginning to end, we can help.
38:26
And the thing about that middle range, I think you're right.
38:31
That's the stuff that makes the stereotype that research is slow, and a lot of times it's also because it's just not pointed enough.
38:35
The researcher can also say in that moment, "I have studied the business plan.
38:40
I know exactly where, I've seen the metrics trend, I have an idea about exactly where that's going to go."
38:47
We still need to do that middle range research.
38:52
The question is valuable, but it's now very pointed and the time is worth it. Amazing.
38:58
Okay, I want to cure the rest of these tropes.
39:01
Okay, research is too slow is the first one.
39:01
The second one, I can do my own research.
39:01
Why do I need researchers?
39:06
And that's true, as product people, I hope you are engaging with customers and listening well.
39:14
But no offense, garbage in, garbage out.
39:14
The thing is, anyone can talk to a user.
39:22
That does not constitute research or insights work because one user can be powerful, but one user can be idiosyncratic.
39:30
And a researcher knows how to get to the heart of that really quick.
39:35
They know how to take that conversation, and understand, and situate it in a way which means like, "Sure, democratize research. That's happening.
39:41
There are tools out there that will let anybody get customer feedback, voice a customer type stuff."
39:46
But a researcher is there to help you turn garbage into something that's not garbage and avoid the bias that can come from you just reaching out to your cousin's family and then doing whatever they thought you should do to the product.
40:04
So that's the second trope.
40:04
The third one is AB test everything.
40:04
And AB tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that flipped a stat sig, and then they're like, "Cool, I was significantly down over this course of time for these users."
40:29
And then they just start speculating about why that is, because the AB test rarely tells you why it changed in the way it did.
40:36
And then this endless flywheel of AB testing goes and I'm like, "Hey, you don't have to guess.
40:43
I know somebody who can get you an answer or at least evidence that addresses the question of why did we see the test result we did in a very short amount of time?
40:53
Or you could use your customers as Guinea pigs, and throw more experiments at them over and over, and spend a long time on it, and come to the same place in the end."
41:03
I think a similar critique that PMs often have is AB testing is conclusive scientifically, statistically, user research is just talking to a bunch of people. Why would I trust that?
41:10
What is your best way to help PMs realize that this is actually very valuable data and you should listen to it?
41:23
It's not just, you know, a story here and there. Yeah.
41:26
No, I think they're both right.
41:26
AB testing is as close as we can get to making causal claims about products.
41:33
Research is usually not oriented towards making causal claims or it should not be, but those causal claims rarely tell you how and why things happen.
41:39
And if you want to not make that mistake again in the future, you need to know how and why.
41:45
If you want to build a better product in a way that doesn't just answer this narrow question that an AB test answered, you need to know how and why. And so you need both.
41:54
Beautiful partnerships between data scientists and research and insights people are, I think what we're going to see in that next evolution.
42:05
And if you set that virtuous cycle up, if you set up the engagement where those people are involved from the beginning, you don't make those mistakes.
42:10
You get the causal relationship, which is valuable for one reason and the hows and whys, which are valuable for other reasons. Awesome. Okay.
42:19
I think there's two more tropes you had.
42:23
One of them is a simple one, which is like everyone loves to quote that it turns out a totally apocryphal Henry Ford quote about, "If I'd asked my users."
42:27
It turns out to the best of our knowledge, he did not say that. And- Really? What? Yeah, I know. Isn't that sad? I didn't know that. I know.
42:38
Sorry to burst your bubble, Lenny. Oh, wow.
42:41
Who was- Does anyone say anything?
42:42
I feel like every quote is- Is apocryphal, now? I know. Yeah. What is reality? Geez, can we?
42:46
Well, let's just- Okay, maybe he said that.
42:49
He certainly believed that.
42:49
That's what the historians say.
42:49
But the reason that makes researchers so angry is because that's not research.
42:53
That's not what researchers do.
42:59
A researcher who's going to ask customers what they want is a bad researcher.
43:04
You need a different researcher.
43:04
I've never done that in my career.
43:04
No one on my team has ever run a study that's like that.
43:09
So that just makes researchers mad.
43:15
And then the last one is about post-hoc bias.
43:15
It's, "We knew this already. That was obvious."
43:23
And I think a lot about this book, which I would recommend to your listeners.
43:23
The author is a sociologist at UPenn named Duncan Watts, and the title is Everything is Obvious If You Already Know the Answer.
43:36
And it's about hindsight bias.
43:36
He makes the argument that we rely too much on intuition, heuristics, and pattern matching in a way that is inappropriate to our experience.
43:49
And it's like it leads us astray.
43:49
It's like a form of self gaslighting.
43:49
And it happens because we end up selectively remembering things and then constructing narratives around them in a way which makes us feel like we already knew that, when we in fact did not.
44:04
And he talks about this other, one of those cognitive biases called the narrative fallacy, which is the idea that people love to make convenient, simple stories about the past.
44:15
If I asked you about your career, Lenny, and how you got to be this amazing podcast host, you'd be like, "Well, let me tell you about this series of events." And we do that.
44:19
It's part of how we make sense of our lives and the information around us, but it would probably be a lie in the sense that we all twist the evidence we have to fit the narrative we want to be true, because it's simple, and lovely, and makes us happy.
44:35
This is going to sound self-serving, but I find I'm the opposite.
44:37
I'm like, "I have no idea how this all came about.
44:37
Here's some things that happened, and somehow I ended up here."
44:42
But maybe I'm being very modest and try to not give myself any credit. That's beautiful.
44:50
Thank you for these tropes, by the way. This was fun.
44:50
I didn't know you were going to do that.
44:52
So that's a fun, little collection we've got, here. Thanks.
44:55
I wanted to ask about, there's this tweet by Patrick Collison that I've brought up a couple of times on this podcast, that I think is really interesting.
45:01
And his tweet is this, "In my opinion, the best product will stem from a very strong mental model of the domain and user.
45:08
User research can help you get such a model and validate it along the way.
45:14
But it's important to view the syllogism of UXR as model of user research, to improving your mental model of the user, to what product you should build versus user research tells you what product to build."
45:25
Does that resonate in any way thoughts on that way of thinking about user research?
45:32
Yeah, there's a double-edged sword we talk about a lot in the research community, which is about making recommendations for design.
45:46
So the best research doesn't leave it at that.
45:46
It tells you, and it's like the what, the so what, and then the then what.
45:54
But the problem with that is some researchers go too far in the other direction, where they're like, "We ran this study, it yielded these insights, and therefore this is what we should build."
46:04
And everyone else on the team is like, "Whoa, whoa, whoa.
46:09
Glad to hear your thoughts on the matter, but there's a lot going on here.
46:09
Maybe we should talk about it."
46:15
And that makes perfect sense.
46:15
That's a failure of communication.
46:19
And I think that speaks to the thing that Patrick is saying, is like, "Good research can sometimes tell us exactly what the problem is and exactly how to fix it."
46:24
An example of that is the multimillion dollar button I told you about.
46:29
But in a lot of the bigger picture questions, especially the macro ones and maybe also the really pointed middle range ones, the point isn't really, "This is exactly what we should do and this is exactly what we should build."
46:43
It is, "Let us develop a framework which is based on actual evidence, and then together as a team figure out how we want to experiment our way to a successful product."
46:53
To close the loop on this specific thread, what is your advice to teams, researchers to help move out of this reckoning, and to move forward, and help the field, both from a user researcher perspective and also from just a company that maybe laid off a bunch of user researchers or is trying to decide what to do with their researchers? Thank you for asking.
47:12
I think I said to you earlier, and I feel some pressure as maybe the first conversation that you've had specifically about research on this podcast. Yeah, I think so. And I want to help.
47:25
I believe so much in this discipline of research and insights, and I think when I said, "The UX research discipline of the last 15 years is dying," I didn't mean that I think research is dying, far from it.
47:36
I think that there's a version of it, which we're now moving past and into a new version.
47:42
We're going through an evolution, as many do.
47:45
And so the question for me is like, "How can researchers, and the companies, and the other people with whom they work create a new version, a different version, an evolution, which is hugely impactful for the business?"
47:55
And so the advice I'd give to researchers about that is develop diverse research skills.
47:59
Remembering the five or five and a half tool list that I mentioned earlier, really go deep on that business knowledge, so speaking the language of product, and business, and metrics, and understanding exactly how to use your insights like a scalpel, building those strong relationships, which is not a thing that researchers can do by themselves.
48:23
It requires two-way engagements, and also in a way which allows researchers to do fewer things better.
48:28
So most researchers that I know are working on teams where they're like, "I'm the only researcher, and I have seven PMs and 20 designers, and I'm trying to do 10 projects."
48:37
And no one's going to do a good job that way.
48:42
So researchers have to learn with their partners about how to say no and focus on the most important things.
48:46
But that's only half of it, right?
48:46
That's the research side.
48:51
I have two thoughts about what companies should be doing.
48:51
The first one, it's a little bit of an aside, but not really.
48:58
One thing I learned by through the responses to the article was everybody came out of the woodworks from the variety of insights disciplines that are out there.
49:08
Because I come from a tradition of user experience research or user research, but there are many insights disciplines in many industries, and they all wanted to claim one type of research or another, and say, "Oh, well, we overhear in consumer insights or market research have been doing that well for years."
49:25
And there are many insights disciplines.
49:25
And generally I think creating silos is stupid.
49:30
Actually, I'm curious what you think, because here's the number one thing I heard when I joined Airbnb and you were there, is I did it a quick listening tour where I talked to a bunch of product people.
49:38
And they all said the same thing.
49:43
They were like, "Listen, we have all these different people throwing insights over the transom. And it's great.
49:48
We want to hear from the data scientists, from the product specialists, from the customer service people, and the voice of the customer, whatever, all that stuff.
49:57
But they're all coming over the side and we don't know what to make of it. It's too much."
50:03
And that, as much as anything, is an argument for companies to stop siloing research disciplines.
50:08
So when I joined Airbnb, I set out to create an integrated insights function where it's like, "Let's do UX research, let's talk about the market and competitors when we have to.
50:13
Let's integrate smartly with data science functions.
50:18
Let's integrate all the stuff we're getting from customer service feedback."
50:23
We brought over what was then the NPS program and said, "Hey, if we're getting customer feedback there, let's all just use it all to fuel this one insights machine."
50:33
So that's the first piece of advice I'd give companies.
50:37
And the second one, without being a broken record, is to think differently about the broken cycle.
50:37
So integrate researchers into a unified, lean process.
50:42
integrate researchers into a unified, lean process. So if the researcher is not there from beginning to end, if there are not strong relationships between product people and design
50:54
people at every level, engineering people at every level, and somebody who's their insights partner, we're going to fall back into this problem where we're just a service discipline, we're not extracting the maximum value, it comes too late, we don't know what questions to ask, we're ignorant about what research can do. And so creating that integrated, lean process where
51:08
And so creating that integrated, lean process where a researcher is arm in arm from the beginning, is the most important advice I'd give.
51:19
That last piece may be the answer to this next question, but the question is how can product managers be better partners to user researchers/get more leverage out of user researchers?
51:29
I think that is in many ways the answer, making sure that they are creating a process for the product, for their products, that it integrates user researchers and insights from beginning to end.
51:38
Also, being willing to partner with the research on the roofless prioritization.
51:43
I used to say that, "A full plate for a researcher was probably three things, two big projects and a small project, like a side project.
51:55
More than that, your researcher is probably not doing a very good job.
51:58
And a project may take 48 hours. That's okay.
51:58
But so they need your help to prioritize, they need you to participate.
52:04
Great PMs will take the time to be with researchers to go into the field, even to travel."
52:10
Did you ever do that, Lenny? I did.
52:15
I went with Louise, who introduced, we came up with this, basically told me to chat with you about this topic. Thanks, Louise. Thanks, Louise.
52:24
We did a whole tour to Paris, our whole team, or the leads of our team went to Paris to do a bunch of focus groups and a bunch of user research behind actual mirrors.
52:34
I've never done that before that trip, and it was amazing. We learned a ton.
52:38
Can I tell you a quick story about behind the mirror? Please.
52:40
This is back from when I was at Facebook.
52:41
And there was the high times there, it was like 2012, '13, and newsfeed is really taking off, ads are going into newsfeed.
52:49
And I was a leader of a team that was working among other things on how to address post quality.
52:55
Like, "How do we think about what's a good post and how do we get feedback about it?"
53:01
And so there was a team of engineers that thought, "One thing that you can do on Facebook is hide a post."
53:06
So they were like, "This is easy.
53:06
Let's look at the posts that are hidden the most and use that as the signal of what's a good post on Facebook?" Seems reasonable.
53:19
And something tripped me on this one. And so I did two things.
53:26
So the first thing I did is I looked at the distribution of hiding by user, and found out that it's power law distributed, like everything on the internet.
53:32
There are a few people on Facebook who hide a ton, and then most people don't hide at all.
53:38
And so then what we did was, we call these super hiders, we called them super hiders.
53:46
And so we said, "Let's find super hiders around the office, and we'll get a super hider in, and we'll do a really traditional user interview." We just wanted to see.
53:58
So literally the first person who walked in, I remember, because this was a person who had those fingernails that are so long, you don't know how they can do touchscreens, but they did. They were amazing at it.
54:09
And it was one of those rooms with the glass.
54:09
And I insisted that the ENG directors, the product people, and they were willing, whatever.
54:20
So everybody's behind the glass, and I'm there with them, and the excellent researcher is in the room, and they come in, and we're just doing a traditional think aloud study.
54:24
And so they go, "Hey, can you open up your Facebook app?
54:28
We would just love to see what your experience is like."
54:32
So they open up Facebook, and were looking, and they look at the first story, and they hide it.
54:36
They go to the second story and they hide it.
54:36
And this went on for a while.
54:43
And she's definitely using Facebook, but every time she'd finish with a story, she'd hide it.
54:47
And the people in the back room were starting to chatter.
54:47
And they're like, "Wait, what?
54:53
What is happening right now?"
54:53
And like the good researcher that this person was, they let it continue, and they're like, "Whoa, can you tell me what you're thinking right now?"
54:59
Come to find out that she was like, "Well, I hid that story because I'd seen it already."
55:03
The model she was going for was inbox zero, which was sad, because it was infinitely scrolling.
55:10
She would never get there.
55:16
And the reason I liked that story is because the people in the back room had their minds blown.
55:23
It was not that we assumed that was common behavior, like this person could have been unique, but it was enough, because those people were there, experiencing the research, that N of one allowed them to burst their own bubble and realize, "Okay, we can't think so naïvely about hides as a signal anymore."
55:42
And we came up with a better solution.
55:47
That is an awesome story and such a good example of you don't need statistical significance to get massive insights.
55:52
One example just gives you a, "Wow, this might be exactly what's happening. Let's go validate that."
55:58
Versus, like, "We are confident, 100%, this is what happened." I love that.
56:04
It reminds me actually in the mirror study that I was talking about in Paris, there's a Facebook element to it, too.
56:08
We were trying to convince hosts how to feel more comfortable accepting guests who are booking instantly.
56:12
And one of our theories was if they were connected on Facebook, they would be more comfortable letting someone book instantly. Yeah.
56:21
And we're just like, "Hey, what if you were to connect Facebook and see if they're friends?"
56:22
And everybody in Paris was very afraid of connecting and giving Facebook any data, way ahead of what the US hosts were feeling. Yeah.
56:34
So it just made it very clear nobody wants to actually give Facebook any data.
56:36
So it was very anti-Facebook at that point.