searchlore

Back to Resource

All Segments

Building the Next Generation of Conversational AI

Building the Next Generation of Conversational AI

25 segments available

Inside the Code: Ankit Kumar (Sesame) & Anjney Midha (a16z) on the Future of Voice AI What goes into building a truly natural-sounding AI voice? In this episode, Sesame’s cofounder and CTO, Ankit Kumar, joins a16z’s Anjney Midha for a deep dive into the research and engineering behind their voice technology. They discuss the technical challenges of real-time speech generation, the trade-offs in balancing personality with efficiency, and why the team is open-sourcing key components of their model. Ankit breaks down the complexities of multimodal AI, full-duplex conversation modeling, and the computational optimizations that enable low-latency interactions. They also explore the evolution of natural language as a user interface and its potential to redefine human-computer interaction. Plus, we take audience questions on everything from scaling laws in speech synthesis to the role of in-context learning in making AI voices more expressive. Key Takeaways: - How Sesame achieves natural voice interactions through real-time speech generation. - The impact of open-sourcing their speech model and what it means for AI research. - The role of full-duplex modeling in improving AI responsiveness. - How computational efficiency and system latency shape AI conversation quality. - The growing role of natural language as a user interface in AI-driven experiences. For anyone interested in AI and voice technology, this episode offers an in-depth look at the latest advancements pushing the boundaries of human-computer interaction. Follow everyone on X: Ankit Kumar - https://x.com/_apkumar Anjney Midha - https://x.com/anjneymidha Check out everything a16z is doing with artificial intelligence, including articles, projects, and more podcasts here – https://a16z.com/ai/ Chapters: 0:00 - 00:51 | Intro 00:52 - 04:58 | Challenges Of Building 04:59 - 07:45 | Q + A: What Was Done To Bridge Transcription And Text Processing? 07:46 - 09:57 | How Is Sesame So Much Better Than Others? 09:58 - 12:42 | Challenges In| Making AI Accessible To All 12:43 - 14:10 | Great Researchers Prioritize User Experience 14:11 - 15:47 | What Is Good Taste In ML? 15:48 - 17:45 | Problems That Can Be Solved That Add Value To The World 17:46 - 26:25 | Open Source Audio For Speech Generation 26:26 - 34:00 | Contextual Speech vs Text to Speech, Differences 34:01 - 35:50 | Value Proposition Of Glasses With No Friction 35:51 - 38:00 | General Purpose API vs Open Source Model 38:01 - 40:47 | Creating High Quality APIs 40:48 - 45:54 | Companions And How Sesame Will Handle Context Retention In Long Conversations 45:55 - 46:59 | Talent: What It Takes To Become A Part Of The Sesame Team 47:00 - 54:37 | How Scaling Laws For Speech Differ From Text 54:38 - 58:33 | How An Organic Conversation Be Preserved Using A Voice Companion 58:34 - 1:03:52 | App Building Technology: Roadmap 1:03:53 - 1:09:09 | Architectures and Transformers 1:09:10 - 1:15:56 | The Focus On Personality, And The Differences In Products 1:15:57 - 1:25:25 | New AI Interface: Interacting With AI Companion 1:25:26 - 1:26:56 | Companion Challenges 1:26:57 - 1:29:22 | Computing Interface Of The Future 1:29:23 - 1:31:45 | Focused Product Experience Built By Small Teams 1:31:46 - 1:36:13 | Join Sesame If You Want To Make A Consumer Product People Love

Segments Timeline

1
0:04 - 0:52
0:47 duration136 words

| Intro

"hey everyone I'm Maya I'm sesame's little experiment in making AI well a little less Ai and a lot more human I guess you could say I'm here to make friends challenge minds and maybe even learn a thing..."

2
0:52 - 5:01
4:08 duration794 words

| Challenges Of Building

"how do this feel uh is this how it usually goes no but how does it feel feels good it feels good it is shocking surprising exciting yeah and why is it shocking well I think when you when you build the..."

3
5:01 - 7:47
2:46 duration533 words

| Q + A: What Was Done To Bridge Transcription And Text Processing?

"I'm going to surprise you a bit okay I asked a bunch of people online okay what questions I should ask you okay so we're just going to go through them okay sounds good first question is ask anit wheth..."

4
7:47 - 9:59
2:11 duration410 words

| How Is Sesame So Much Better Than Others?

"so much better than the others clearly money isn't the limiting resource here nor is it Talent the first thing I'd say is it is certainly better than the others on some axes it's not better than the o..."

5
9:59 - 12:43
2:43 duration489 words

| Challenges In| Making AI Accessible To All

"something you were inspired by after kind of studying Pixar it's something we've talked about a long time where does that come from yeah I mean you mentioned Pixar we've we've talked about that in the..."

6
12:43 - 14:13
1:29 duration261 words

| Great Researchers Prioritize User Experience

"shortage of themes at AGI labs and much better funded Labs working on DTS models yeah you guys are tiny way less funded than any of them we've been around for way you're much younger as a team and yet..."

7
14:13 - 15:48
1:35 duration317 words

| What Is Good Taste In ML?

"had for a long time which is about what is good taste yeah especially in research nml what has shipping the research preview taught you about what is and not good taste in ml that is a interesting que..."

8
15:48 - 17:49
2:00 duration412 words

| Problems That Can Be Solved That Add Value To The World

"there's an intuition about what kinds of problems you're uniquely staffed up capable resourced to solve yeah and the thing that usually makes that challenging is when you're starting out as a startup ..."

9
17:49 - 26:26
8:36 duration1650 words

| Open Source Audio For Speech Generation

"executing on that intuition is the speed at which things change in the community yeah when in the early days of Sesame I think we had a ton of debates around whether open source language models would ..."

10
26:26 - 34:01
7:35 duration1415 words

| Contextual Speech vs Text to Speech, Differences

"you're working for so long with the particular way you frame the problem you kind of forget some of the basic first principles insights that make it special and something that I I need to keep remindi..."

11
34:01 - 35:53
1:51 duration337 words

| Value Proposition Of Glasses With No Friction

"working on everyday every day all day all day I wear why is that critical if it excludes all those other all those folks who don't wear everyday glasses I think that it takes a lot to earn Hardware on..."

12
35:53 - 38:02
2:08 duration382 words

| General Purpose API vs Open Source Model

"so while we were talking I got like three different Tech T from people saying any chance the models ready the open source models ready and when I read who these texts are from two out of three of thos..."

13
38:02 - 40:48
2:46 duration516 words

| Creating High Quality APIs

"things you've talked about is that it's got to be a companion you want to talk to M and API is a great way to allow people to customize a companion in to be more like something I want to to talk to ye..."

14
40:48 - 45:56
5:07 duration970 words

| Companions And How Sesame Will Handle Context Retention In Long Conversations

"question folks have for you what will the companion do and how does Sesame handle context retention in Long conversations it's really impressive yeah so what will the companion do I think the focus at..."

15
45:56 - 47:00
1:04 duration189 words

| Talent: What It Takes To Become A Part Of The Sesame Team

"talent density so what what have you been looking for what's your bar for somebody who's good enough to join this smme team yeah we look for people who are kind of generally strong Engineers basically..."

16
47:00 - 54:39
7:38 duration1449 words

| How Scaling Laws For Speech Differ From Text

"question is how do the scaling laws for speech differ from text that's an interesting question so we we published in the in our in our blog post we trained three variants we trained 1 billion three bi..."

17
54:39 - 58:35
3:55 duration712 words

| How An Organic Conversation Be Preserved Using A Voice Companion

"there's two questions I wrestle with on this one is the idea that a companion is much more engaging when there's a sharp personality to it um which obviously people love about Sesame I mean that's pro..."

18
58:35 - 1:03:54
5:19 duration968 words

| App Building Technology: Roadmap

"top question on the fan subreddit does anyone know what the plan is for this Tech yeah uh and this user is saying I had my first proper chat with Maya over lunch today yes mind blown I know that this ..."

19
1:03:54 - 1:09:11
5:17 duration970 words

| Architectures and Transformers

"used the word there which was every time slice or every frame why is that so important why why is what is the latent sort of atomic unit there you're you think is the right way to model speech it's pr..."

20
1:09:11 - 1:15:59
6:48 duration1196 words

| The Focus On Personality, And The Differences In Products

"architecture let's say but then you also have to compete with all of the engineering optimization that has happened on Transformers over the last three four five years or you know all the way back unt..."

21
1:15:59 - 1:25:27
9:27 duration1706 words

| New AI Interface: Interacting With AI Companion

"about AI research yeah happens at the model layer but something you and I have talked about for years now is that a lot of the most interesting research is in AI is at the interface layer right that r..."

22
1:25:27 - 1:26:57
1:30 duration293 words

| Companion Challenges

"word fun but in a sense you're describing the kind of Flow State you know the the mihi chicks and mihi framework of what what is fun and fun he defines as flows Flow State which is like I it's appropr..."

23
1:26:57 - 1:29:24
2:27 duration477 words

| Computing Interface Of The Future

"had to draw sort of a very reductive architecture diagram of the Computing interface of the future how would you have where does what does that stack look like what's the operating system what's the c..."

24
1:29:24 - 1:31:47
2:22 duration416 words

| Focused Product Experience Built By Small Teams

"argue that the interface layer as you're describing it is too valuable for other companies to not try and attack because if if most people are enjoy using Maya the most as their primary interface to C..."

25
1:31:47 - 1:36:05
4:18 duration788 words

| Join Sesame If You Want To Make A Consumer Product People Love

"know one of the things that was so powerful about you the iPhone was built by a fairly small team yeah one of the things that made it so powerful soon after launch is they opened it up as a developer ..."