searchlore

Back to Resource

All Segments

Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

Emmett Shear on Building AI That Actually Cares: Beyond Control and Steering

15 segments available

Emmett Shear, founder of Twitch and former OpenAI interim CEO, challenges the fundamental assumptions driving AGI development. In this conversation with Erik Torenberg and Séb Krier, Shear argues that the entire "control and steering" paradigm for AI alignment is fatally flawed. Instead, he proposes "organic alignment" - teaching AI systems to genuinely care about humans the way we naturally do. The discussion explores why treating AGI as a tool rather than a potential being could be catastrophic, how current chatbots act as "narcissistic mirrors," and why the only sustainable path forward is creating AI that can say no to harmful requests. Shear shares his technical approach through multi-agent simulations at his new company Softmax, and offers a surprisingly hopeful vision of humans and AI as collaborative teammates - if we can get the alignment right. Timecodes: 00:00 - “If it's a machine, it's a tool. And if it's a being, it's a slave” 01:01 - Alignment Takes an Argument: The Hidden Assumptions in "Aligned AI" 02:26 - Alignment as Process, Not Destination 04:23 - Morality as Ongoing Learning, Not Fixed Rules 08:09 - Most AI Alignment Is Actually Steering (Or Slavery) 09:01 - The Dangerous Assumption: "We're Making Beings, But They Don't Count" 14:37 - Goal Inference and Theory of Mind 23:01 - The Foundation of Care 24:41 - Why Most AI Labs Focus on Steering and Control 27:35 - The Only Good Outcome: A Being That Actually Cares About Us 32:42 - The Substrate Question: Does Silicon vs. Carbon Matter? 51:24 - The Only Sustainable Form of Alignment 54:55 - AI Chatbots and Social Dynamics 59:50 - AI Futures: Tools, Beings, and Society 1:01:54 - Visions for a Good AI Future Resources: Follow Emmett on X: https://x.com/eshear Follow Séb on X: https://x.com/sebkrier Follow Erik on X: https://x.com/eriktorenberg Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://x.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details, please see a16z.com/disclosures.

Segments Timeline

1
0:00 - 1:01
1:01 duration225 words

“If it's a machine, it's a tool. And if it's a being, it's a slave”

"Most of the AI is focused on alignment as steering. That's the plight word. If you think that they were making our beings, you'd also call this slavery. Someone who who you steer, who doesn't get to s..."

2
1:01 - 2:26
1:24 duration321 words

Alignment Takes an Argument: The Hidden Assumptions in "Aligned AI"

">> When people think about alignment, I think there's a lot of confusion. People talk about things being aligned. We need to build an aligned AI. And the problem with that is when someone says that it..."

3
2:26 - 4:23
1:57 duration364 words

Alignment as Process, Not Destination

"to. And so when we talk about organic alignment um I think the important thing to to recognize is that alignment is not a thing. It's not a state it's a process. And like this is this is one of these ..."

4
4:23 - 8:09
3:45 duration693 words

Morality as Ongoing Learning, Not Fixed Rules

"mean. And it's like this will be act as a morally good being. And acting as a morally good being is a process and not a destination. We don't we never unfortunately we've we've tried taking down table..."

5
8:09 - 9:01
0:52 duration182 words

Most AI Alignment Is Actually Steering (Or Slavery)

"if you have a child that only only follows the rules, that's not a moral person that you've raised. is you raise a dangerous person actually who will probably do great harm following the rules. And if..."

6
9:01 - 14:37
5:35 duration1341 words

The Dangerous Assumption: "We're Making Beings, But They Don't Count"

">> Yeah. I mean, it's um Yeah. I have a lot of um you know similar intuitions about certain things like um I also dislike the um you know the idea that kind of you know we just need to like crack the ..."

7
14:37 - 23:01
8:24 duration1803 words

Goal Inference and Theory of Mind

"making EMTT important because there's some lossiness between the description and the actual or why is the distinction? It it goes back to my what I was saying like this is you technical alignment is t..."

8
23:01 - 24:43
1:41 duration313 words

The Foundation of Care

"care is not conceptual. Care is non-verbal. It doesn't indicate what to do. It doesn't indicate how to do it. Uh care is a relative waiting over effectively like attention on states to it's a relative..."

9
24:43 - 27:37
2:54 duration584 words

Why Most AI Labs Focus on Steering and Control

"question was just how does this what does this look like in AI systems and maybe another way of asking is like um when you when you talk to the people most focused on alignment at the at the major lab..."

10
27:37 - 32:43
5:05 duration1100 words

The Only Good Outcome: A Being That Actually Cares About Us

"because something that has the general ability to effectively use judgment, think for itself, discern between pro possibilities is obviously a thinking thing. Like and so as you go from what we have t..."

11
32:43 - 51:26
18:43 duration4200 words

The Substrate Question: Does Silicon vs. Carbon Matter?

">> Had to think about it but I think you know it even depends what we mean by person and you know in some sense I care about certain corporations too. So I'm I'm >> No, no, no. I mean but like you car..."

12
51:26 - 54:55
3:28 duration726 words

The Only Sustainable Form of Alignment

"your your strategy of how you're trying to achieve or even attempt to achieve this this this level like in terms of research or road map or >> so uh the in order to be good at tech we're basically foc..."

13
54:55 - 59:50
4:55 duration988 words

AI Chatbots and Social Dynamics

"not they're not beings yet. They don't really have a coherent sense of like self and desire and goals and stuff right now. And so mostly they just pick up on you and reflect it. You know, modulo some ..."

14
59:50 - 1:01:54
2:03 duration428 words

AI Futures: Tools, Beings, and Society

"been optimized around relatively high signal low entropy environments like coding and math which is why they're those are easy or relatively easy. um and like talking to a single person whose goal it ..."

15
1:01:54 - 1:07:32
5:37 duration1173 words

Visions for a Good AI Future

">> Two last questions we'll get you out of here. In as much detail as possible can you explain your what your vision of an AI future actually looks like? Like a good good AI future. >> Yeah. Um, the g..."