All Segments
Al Engineering 101 with Chip Huyen (Nvidia, Stanford, Netflix)
41 segments available
Chip Huyen is a core developer on Nvidia’s Nemo platform, a former AI researcher at Netflix, and taught machine learning at Stanford. She’s a two-time founder and the author of two widely read books on AI, including AI Engineering, which has been the most-read book on the O’Reilly platform since its launch. Unlike many AI commentators, Chip has built multiple successful AI products and platforms and works directly with enterprises on their AI strategies, giving her unique visibility into what’s actually happening inside companies building AI products. *We discuss:* 1. What people think makes AI apps better vs. what actually makes AI apps better 2. What pre-training vs. post-training is, and why fine-tuning should be your last resort 3. How RLHF (reinforcement learning from human feedback) actually works 4. Why data quality matters more than which vector database you choose 5. Why high performers are seeing the most gains from AI coding tools 6. Why most AI problems are actually UX issues *Brought to you by:* Dscout—The UX platform to capture insights at every stage: from ideation to production: https://www.dscout.com/ Justworks—The all-in-one HR solution for managing your small business with confidence: https://www.justworks.com Persona—A global leader in digital identity verification: https://withpersona.com/lenny *Transcript:* https://www.lennysnewsletter.com/p/al-engineering-101-with-chip-huyen *My biggest takeaways (for paid newsletter subscribers):* https://www.lennysnewsletter.com/i/176081814/my-biggest-takeaways-from-this-conversation *Where to find Chip Huyen:* • X: https://x.com/chipro • LinkedIn: https://www.linkedin.com/in/chiphuyen/ • Website: https://huyenchip.com/ • Substack: https://substack.com/@chiphuyen *Where to find Lenny:* • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ *In this episode, we cover:* (00:00) Introduction to Chip Huyen (04:28) Chip’s viral LinkedIn post (07:05) Understanding AI training: pre-training vs. post-training (08:50) Language modeling explained (13:55) The importance of post-training (15:20) Reinforcement learning and human feedback (22:23) The importance of evals in AI development (31:55) Retrieval augmented generation (RAG) explained (38:50) Challenges in AI tool adoption (43:19) Challenges in measuring productivity (45:20) The three-bucket test (49:10) The future of engineering roles (55:31) ML Engineers vs. AI engineers (57:12) Looking forward: the impact of AI (01:05:48) Model capabilities vs. perceived performance (01:08:23) Lightning round and final thoughts *Referenced:* • Chip’s LinkedIn post on what actually improves AI apps: https://www.linkedin.com/posts/chiphuyen_aiapplications-aiengineering-activity-7358971409227792384-y0mf/ • Prediction and Entropy of Printed English: https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf • Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor): https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody •Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO): https://www.lennysnewsletter.com/p/inside-handshake-garrett-lord • First interview with Scale AI’s CEO: $14B Meta deal, what’s working in enterprise AI, and what frontier labs are building next | Jason Droege: https://www.lennysnewsletter.com/p/first-interview-with-scale-ais-ceo-jason-droege • Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next • Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course): https://www.lennysnewsletter.com/p/why-ai-evals-are-the-hottest-new-skill ...References continued at: https://www.lennysnewsletter.com/p/al-engineering-101-with-chip-huyen *Recommended books:* • The Complete Sherlock Holmes: https://www.amazon.com/Complete-Sherlock-Holmes-Volumes/dp/0553328255 • AI Engineering: Building Applications with Foundation Models: https://www.amazon.com/AI-Engineering-Building-Applications-Foundation/dp/1098166302 • The Selfish Gene: https://www.amazon.com/Selfish-Gene-Anniversary-Introduction/dp/0199291152 • From Third World to First: The Singapore Story: 1965-2000: https://www.amazon.com/Third-World-First-Singapore-1965-2000/dp/0060197765 _Production and marketing by https://penname.co/._ _For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com._ Lenny may be an investor in the companies discussed.
Segments Timeline
The Importance of User Feedback in AI Development
Chip Huyen emphasizes the critical role of user feedback in improving AI applications. She discusses how many companies struggle with AI product development despite having advanced tools, highlighting the disconnect between technology and user needs. This segment explores the misconception that staying updated with AI news is essential, arguing instead for a focus on direct user engagement and data quality.
"One question that get asked a lot and a lot is how do we keep up to date with the latest AI news? Why why do you need to keep up to date with the latest AI news? If you talk to the users and understan..."
What Actually Improves AI Apps
In this segment, Chip Huyen contrasts common beliefs about AI app improvement with the reality of effective practices. She shares insights from her viral LinkedIn post, revealing that talking to users, building reliable platforms, and optimizing workflows are far more impactful than merely adopting the latest technologies or frameworks. This discussion sheds light on the fundamental principles of successful AI development.
"our sponsors. This episode is brought to you by Dcout. Design teams today are expected to move fast, but also to get it right. That's where Dout comes in. Dout is the all-in-one research platform buil..."
Pre-Training vs. Post-Training in AI
Chip Huyen breaks down the concepts of pre-training and post-training in AI models. She explains how pre-training involves encoding statistical information about language, while post-training focuses on fine-tuning models for specific tasks. This segment provides clarity on these essential processes, helping listeners understand their significance in AI development.
"apps, staying up to date with the latest AI news, adopting the newest agentic framework, agonizing over vector databases to use, constantly evaluating what model is smarter, fine-tuning a model, and t..."
Understanding Language Modeling
In this segment, Chip Huyen delves into language modeling, explaining how AI models predict the next word in a sequence based on statistical likelihood. She discusses the importance of training on large datasets and how models adjust their predictions through a process of learning from errors. This foundational knowledge is crucial for anyone looking to grasp the mechanics behind AI language models.
"just how fine-tuning fits into that? Just what fine-tuning actually is. >> Disclaimer, I don't have like full visibility into like on what like this big secretive like frontier labs are doing. Uh but ..."
The Role of Tokens in Language Models
Chip Huyen explains the concept of tokens in language models, describing how they serve as a bridge between characters and words. She illustrates how tokens help reduce vocabulary complexity while maintaining the ability to understand new words. This segment highlights the technical intricacies of language modeling and the importance of tokenization in AI applications.
"likely to come back to every color is so so it's just like get is um it's is it's a way of encoding statical information so like when language modeling when you train a large amount of data like it yo..."
The Impact of Sampling Strategies
In this segment, Chip Huyen discusses the significance of sampling strategies in AI language models. She explains how different strategies can influence the creativity and accuracy of model outputs, emphasizing that the choice of sampling can greatly affect performance. This insight is vital for developers looking to optimize their AI applications.
"pick like depending on your sampling strategy like do you want it to always pick the most likely token or you wanted to pick something more creative you know so so so I think sampling strategy I think..."
Supervised vs. Unsupervised Learning
In this segment, Chip Huyen delves into the differences between supervised and unsupervised learning. She explains how supervised learning relies on labeled data to train models, providing examples of companies that specialize in creating high-quality labeled datasets for AI training.
">> this is a good segue to you talked about supervised learning versus unsupervised learning I love we're getting into this by the way this is super interesting so you talked about labeled data basica..."
Reinforcement Learning from Human Feedback
Chip introduces the concept of Reinforcement Learning from Human Feedback (RLHF), explaining how it encourages models to produce better outputs through human comparisons. She discusses the challenges of evaluating model performance and the importance of using human feedback to train reward models.
"learning. I'm not sure if your CEOs that you interview bring up like that term. Uh so so the idea is that um you want people to like so like let's say you have a model give the model like a prop right..."
The Economics of Data Labeling
Chip shares insights on the economics surrounding data labeling companies, discussing the challenges they face due to their dependence on a few major clients. She reflects on the implications for the future of these companies and the competitive landscape in the AI data ecosystem.
"you say just different way of like um collei signals >> awesome yeah that's uh I we had the c of anthropic on the podcast and he talked about their version of RHF which is AIdriven reinforcement learn..."
The Importance of Evals in AI Development
In this segment, Chip Huyen explains the concept of evals in AI, discussing their role in assessing the performance of AI applications. She highlights the creative aspects of designing evals and the necessity of establishing criteria to measure the effectiveness of AI models.
"to me and I'm curious to see how it plays out. >> What I'm hearing is you're uh you're bearish on the future of these data labeling companies because as you said, they don't have a lot of leverage ove..."
Evaluating AI Products: The Vibe Check
Chip discusses the debate around the necessity of evals for AI products, suggesting that while some companies rely on intuition, having a structured evaluation process can lead to better outcomes. She emphasizes the importance of balancing effort between evals and new feature development.
">> we had a whole podcast any vals with HML Haml and Shrea and uh and that's exactly what they talked about is just it's actually really fun to create evals for for companies especially. So let's stil..."
Evaluations: Picking Your Battles
In this segment, Chip Huyen elaborates on the importance of evaluations in AI applications, particularly for core use cases. She suggests that while evals are crucial, companies should prioritize which features to evaluate based on their impact. Huyen shares insights on how to effectively use evals to uncover product opportunities and improve performance, stressing that not every feature requires exhaustive evaluation.
"like maybe like that's a debate is about um I do think that's like a lot of time people just like get things to the to the place when it's like okay good enough people run but and then but of course i..."
Understanding Retrieval-Augmented Generation (RAG)
Chip Huyen introduces the concept of Retrieval-Augmented Generation (RAG), explaining its significance in enhancing AI responses by providing context. She discusses the origins of RAG and how it improves question-answering benchmarks by retrieving relevant information from sources like Wikipedia. Huyen emphasizes the importance of data preparation in RAG implementations, noting that the quality of data directly influences the effectiveness of AI models.
"Hamlin Shrea shared is that people need just like I don't know five or seven evals for the most important elements of their product. Is that is that what you see or do you see a lot more in production..."
Data Preparation for Effective AI
In this segment, Chip Huyen dives deeper into the critical role of data preparation in AI applications, particularly for RAG. She discusses strategies for chunking data effectively and the importance of contextual information for AI models. Huyen shares examples of how rewriting data into question-answering formats can enhance AI performance, highlighting the need for AI-specific documentation that differs from human-readable formats.
"to gather informations you need to do a lot search queries uh you like gathers grab the search results and then from the search results you like uh aggregate and then maybe say okay I'm still missing ..."
The Double-Edged Sword of AI Advances
In this segment, Chip Huyen introduces Persona, a platform focused on identity verification to combat fraud in the age of AI. She highlights the dual nature of AI advancements, where while they offer incredible potential, they also present significant risks, such as identity theft and fraud. This discussion underscores the necessity for robust verification systems in AI applications.
"persona the verified identity platform helping organizations onboard users fight fraud and build trust we talk a lot on this podcast about the amazing advances in AI but this can be a double-edged swo..."
AI Tool Adoption Challenges
Chip shares insights on the challenges companies face when adopting AI tools. She categorizes AI tools into internal productivity and customer-facing applications, discussing how clear outcomes drive adoption. This segment reveals the complexities of integrating AI into existing workflows and the varying levels of success across different organizations.
"spend a little time here because a lot of companies are building AI products. A lot of companies are not having a good time building AI products. Let me ask a few questions along these lines of what y..."
Measuring AI Productivity Gains
Chip Huyen addresses the difficulties in measuring productivity gains from AI tools. She explains how companies often struggle to quantify the impact of AI on their operations, leading to skepticism about its effectiveness. This segment emphasizes the need for better metrics to evaluate AI's contribution to productivity and the varying perceptions among different employee tiers.
"facing um so so customer support chatbot is a big one you have a hotel chain you might have like a booking chatbot which is like somehow massive like a lot of booking chatbot because I guess it's it's..."
The Three-Bucket Test for AI Tools
In this segment, Chip discusses a company's innovative approach to evaluating AI tool effectiveness through a three-bucket test. By categorizing engineers into high, mid, and low performers, they assess the impact of AI tools on productivity. This method provides valuable insights into how different skill levels interact with AI technologies, revealing the nuanced effects of AI assistance.
"adoptions of like tooling. So, internal productivity that's where it gets tricky. I would say like a lot of companies uh what they think of as a strategy like I think of as have like usually have very..."
Senior Engineers and AI Resistance
Chip explores the resistance some senior engineers have towards adopting AI tools. She contrasts experiences from different companies, highlighting how high-performing engineers may benefit more from AI, while others may be skeptical due to high standards. This segment sheds light on the varied responses to AI integration within engineering teams.
"right a lot of companies not using coding agents uh or coding aided coding uh and um I was asking I was like I was like okay do do you think that like it helps with your productivity and a lot of time..."
AI's Role in Engineering Team Dynamics
Chip discusses how AI tools are reshaping engineering team dynamics. She explains how companies are restructuring roles to leverage AI effectively, with senior engineers focusing on oversight and process development while junior engineers handle coding tasks. This segment highlights the evolving nature of engineering roles in the AI era.
"for you. Whereas for executive, you care more about like the um the maybe you have more like business metrics that you care about. So so you actually think about like what actually drive drive product..."
Randomized Trials in AI Tool Evaluation
In this segment, Chip elaborates on the randomized trial approach used by companies to evaluate AI tools. By giving different performance tiers access to AI assistance, they can measure productivity changes. This method provides a structured way to assess the effectiveness of AI tools across varying skill levels.
"biggest boost out of it and then the second group is just like the um the average performing. So so he so his opinion is like okay the highest performing engineers they also more proactive they say kn..."
The Future of Engineering with AI
Chip concludes with insights on the future of engineering roles in the context of AI advancements. She discusses how companies are preparing for a landscape where fewer but highly skilled engineers will oversee AI-driven processes, emphasizing the importance of holistic problem-solving skills. This segment reflects on the transformative impact of AI on engineering practices.
"it cursor or what did they give them access to? It was cursor. >> I think by then it was cursor. >> Okay, cool. And so >> I didn't work with them. This more like a friend company. >> Okay. It's a frie..."
The Role of Senior Engineers in AI
In this segment, Chip Huyen explains the evolving role of senior engineers in the age of AI. She notes that while junior engineers can produce code with AI assistance, senior engineers are crucial for reviewing processes and ensuring quality, thus preparing for a future where fewer but stronger engineers are needed.
">> Yeah. Uh I definitely like really appreciate as you see companies like we appreciate engineers who are um have a good understanding of the whole systems and be able to have good problem solving ski..."
System Thinking in Engineering
Chip emphasizes the importance of system thinking in computer science education. She shares insights from a webinar with her professor, discussing how coding is a means to solve problems rather than an end in itself. The segment highlights the necessity of understanding complex systems in the context of AI and engineering.
"one become a very strong >> right that's right that's right I feel like >> yeah so so I don't know what's the process was thinking about like yeah um >> no one's thinking It's just it's a problem. We ..."
Challenges with AI in Debugging
Chip Huyen shares her personal experiences with AI in debugging, illustrating the limitations of AI when dealing with complex coding tasks. She recounts a frustrating incident while deploying an application, emphasizing the need for engineers to have a holistic understanding of systems to effectively troubleshoot issues.
"like design step-by-step solution to it will always be there um so I think an example of um of like I actually have a lot of issues with like AI for like um in the way of like is debugging so I'm not ..."
Teaching System Thinking to AI
In this thought-provoking segment, Chip discusses the challenge of teaching AI system thinking. She reflects on the need for human experts to create structured approaches for AI to understand complex problem-solving, highlighting the importance of this skill in the future of engineering.
"with clon is trying to focus on fixing things from a very a different component where the issue is from a different component. So I think I think of like okay be understanding like how different compo..."
The Essence of Computer Science Education
Chip Huyen reiterates the core philosophy of computer science education, emphasizing that it's not just about coding languages but about understanding systems and problem-solving. She connects this idea to insights shared by industry leaders about the future of engineering roles.
"very important. >> That's exactly the same insight Brett Taylor shared on the podcast. He's the co-founder of Sierra. He created Google Maps. He was CEO of Salesforce, Quip, a few other things. And I ..."
AI Engineers vs. ML Engineers
Chip clarifies the distinction between AI engineers and ML engineers, explaining that while ML engineers build models, AI engineers utilize existing models to create products. This segment sheds light on the evolving landscape of engineering roles in the AI domain.
"to be perfect because they always be like edge cases. Um but yeah in general I think it's like just like gen like AI as a service like more as a service like when somebody build the models for you and..."
Future of Product Development with AI
In this segment, Chip discusses the anticipated changes in product development over the next few years. She highlights how organizations are restructuring to integrate AI more effectively, emphasizing the importance of collaboration between engineering, product, and marketing teams.
"world of possibilities. >> Oh yeah. It's like now you don't have to time you don't even have to spend time building this AI brain. Now you can just use it to do stuff. Uh such a such an unlock. Okay. ..."
The Evolution of AI Models
Chip shares her perspective on the future of AI models, suggesting that while base model improvements may plateau, significant advancements will occur in post-training and application building. She discusses the potential for multimodal applications and the challenges associated with voice AI.
"organization just like move fast um so so yeah so I think one big change I see is just like in organizational structure um I think it's like a lot of value place um in like um so before like we have l..."
Voice AI and Human Interaction
Chip explores the complexities of voice AI, discussing the differences between text and voice interactions. She raises important questions about user experience, natural conversation flow, and the ethical implications of AI mimicking human communication.
"mind-blowing it was in the last three years uh so so I think it's like a lot of like improvements we're going to see in the post training phase in the application building phase um and um and yes also..."
Balancing Pre-Training and Post-Training
Chip Huyen shares insights on the balance between pre-training and post-training in AI model development. She discusses the concept of 'test time compute' and how allocating resources for inference can lead to better performance. This segment emphasizes the importance of refining AI models through thoughtful post-training strategies to enhance user experience.
"of course it can be an AI challenge because people are trying to build like voicetovoice model. So instead of having like having to first like transcribe the voice from me into text and then get a mod..."
The Shift Towards Multimodal Experiences
Chip Huyen predicts a significant shift towards multimodal AI experiences, where different functions and roles in engineering will blur. She discusses the automation of work through AI tools and the implications for productivity. This segment highlights the evolving landscape of AI and its impact on various job functions.
"more often versus GPT3 came out like a year I don't know a before after JPT2. So uh maybe true maybe not. And then the fourth point you made is this idea of multimodal investing in multimodal experien..."
Understanding Model Capabilities vs. Perceived Performance
In this segment, Chip Huyen explains the difference between a model's capabilities and its perceived performance. She discusses the importance of generating multiple answers during inference to improve accuracy and user satisfaction. This insight sheds light on how AI can be optimized for better decision-making and reasoning.
"compute is like crazy varies different between different lab um um and also like since then has to spend comput uh on like Jerry inference when I have a train and 500 model now it want to like serve t..."
Fostering Innovation in AI Development
Chip Huyen addresses the current 'idea crisis' in AI development, where many talented individuals struggle to come up with innovative solutions. She shares strategies for fostering creativity and innovation within teams, emphasizing the importance of identifying frustrations and building tools to address them. This segment provides practical advice for leveraging AI to solve real-world problems.
"not change. >> Awesome. >> Does it make sense? Yes. >> Yes. Absolutely. Uh that is a good correlary to uh to Ben man's point. >> Yeah. Chip, we covered a lot of ground. I've gone through everything I ..."
Lessons from Singapore's Transformation
Chip Huyen discusses the impactful leadership of Lee Kuan Yew, the father of modern Singapore, and his book 'From Third World to First'. She highlights how his policies transformed Singapore in just 25 years, emphasizing the importance of public policy and system thinking in nation-building. This segment provides insights into how effective governance can lead to significant societal advancements.
"you have some ideas out there and then it's like last for a long time. That's the one you like live on. I know it's like it's a little bit like um abstract but I thought it's very interesting. The oth..."
The Power of Storytelling in Writing
In this segment, Chip shares her journey as a writer and the importance of understanding audience reactions. She emphasizes the emotional journey in storytelling, discussing how character likability can impact reader engagement. Chip reflects on her experiences with technical writing versus creative writing, revealing the nuances of crafting compelling narratives.
">> What was the name of that second book? >> Uh it's called like from third to first world fashion. I think I have it somewhere here. Yeah, >> there it is. Show and tell. That's that's awesome. I defi..."
Embracing Life's Impermanence
Chip reflects on a life motto that emphasizes the transient nature of existence, suggesting that in the grand scheme, nothing truly matters. This perspective, while seemingly nihilistic, is liberating, allowing her to take risks and embrace challenges without fear of failure. She shares a personal story about loss, reinforcing the idea that material concerns fade in the face of life's ultimate realities.
"it. So I'm interesting like what makes So it's a drama. It's not a science fictions or uh anything that like tech people usually read. So it's it's very like I know it's a very um out of the left fiel..."
The Emotional Journey of Writing
Chip discusses the importance of emotional engagement in writing, particularly in fiction. She explains how understanding the emotional journey of characters can enhance relatability and reader connection. This segment highlights the shift from technical writing to creative storytelling, emphasizing the need for vulnerability and character depth in narrative construction.
"allows me okay let's just try things out right like why does it matter and there a story of like recently um so we have some family member who passed away recently and I was talking to my that because..."
Crafting Relatable Characters
In this segment, Chip shares her insights on character development in storytelling. She recounts feedback on her writing that emphasized the need for likable characters, illustrating how vulnerability can make characters more relatable. This discussion underscores the balance between rationality and emotional depth in creating compelling narratives.
"matters so in a way it's like it's quite liberating >> uh I know you said it might be nihilistic this is what Steve Jobs shared too in one of his most famous speeches just we will all die someday so d..."
Connecting with Readers: Chip's Future Plans
Chip shares her plans to engage with readers through a Substack and YouTube channel focused on system thinking and book reviews. She invites listeners to share books that have influenced their thinking, highlighting her desire to foster a community around thoughtful discourse. This segment encapsulates her commitment to continuous learning and sharing knowledge with others.
"emotional journey was from my editor, right? So like when we write something we we care about like how users would feel like across the the story like we want something in the beginning, right? We wan..."