All Segments
Sergey Levine: Robotics and Machine Learning | Lex Fridman Podcast #108
51 segments available
Sergey Levine is a professor at Berkeley and a world-class researcher in deep learning, reinforcement learning, robotics, and computer vision, including the development of algorithms for end-to-end training of neural network policies that combine perception and control, scalable algorithms for inverse reinforcement learning, and deep RL algorithms. Support this podcast by signing up with these sponsors: - ExpressVPN at https://www.expressvpn.com/lexpod - Cash App - use code "LexPodcast" and download: - Cash App (App Store): https://apple.co/2sPrUHe - Cash App (Google Play): https://bit.ly/2MlvP5w EPISODE LINKS: Sergey's Twitter: https://twitter.com/svlevine Sergey's Website: http://rail.eecs.berkeley.edu/ Sergey's Papers: https://scholar.google.com/citations?user=8R35rCwAAAAJ PODCAST INFO: Podcast website: https://lexfridman.com/podcast Apple Podcasts: https://apple.co/2lwqZIr Spotify: https://spoti.fi/2nEwCF8 RSS: https://lexfridman.com/feed/podcast/ Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4 Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41 OUTLINE: 0:00 - Introduction 3:05 - State-of-the-art robots vs humans 16:13 - Robotics may help us understand intelligence 22:49 - End-to-end learning in robotics 27:01 - Canonical problem in robotics 31:44 - Commonsense reasoning in robotics 34:41 - Can we solve robotics through learning? 44:55 - What is reinforcement learning? 1:06:36 - Tesla Autopilot 1:08:15 - Simulation in reinforcement learning 1:13:46 - Can we learn gravity from data? 1:16:03 - Self-play 1:17:39 - Reward functions 1:27:01 - Bitter lesson by Rich Sutton 1:32:13 - Advice for students interesting in AI 1:33:55 - Meaning of life CONNECT: - Subscribe to this YouTube channel - Twitter: https://twitter.com/lexfridman - LinkedIn: https://www.linkedin.com/in/lexfridman - Facebook: https://www.facebook.com/LexFridmanPage - Instagram: https://www.instagram.com/lexfridman - Medium: https://medium.com/@lexfridman - Support on Patreon: https://www.patreon.com/lexfridman
Segments Timeline
Introduction to Sergey Levine
In this segment, Lex Fridman introduces Sergey Levine, a renowned professor at Berkeley and expert in deep learning, reinforcement learning, and robotics. The discussion highlights Sergey's contributions to the development of algorithms for neural network training and the significance of his research in the field of robotics and AI.
"the following is a conversation with Sergey Levine a professor at Berkeley and a world-class researcher in deep learning reinforcement learning robotics and computer vision including the development o..."
Robots vs Humans: Understanding Capabilities
Sergey Levine discusses the differences between state-of-the-art robots and humans, emphasizing the complexities of robot capabilities. He illustrates the gap between physical hardware and autonomous capabilities, using the example of the PR1 robot, which, despite its impressive tasks, is ultimately controlled by a human.
"now here's my conversation sergey Lavigne what's the difference between a state-of-the-art human such as you and I well I don't know if we qualify Stata they're humans but a state-of-the-art human and..."
The Intelligence Gap in Robotics
In this segment, Sergey explores the significant intelligence gap between humans and robots. He explains how the unpredictability of real-world environments exacerbates this gap, highlighting the challenges robots face when operating outside controlled settings.
"bit on the philosophical questions but how much on the human side of the cognitive abilities in your sense is nature versus nurture so so how much of it is product of evolution and how much of it some..."
Nature vs. Nurture in AI Development
Sergey discusses the implications of nature versus nurture in cognitive abilities, particularly in relation to AI. He reflects on how human adaptability and learning from experience can inform the development of intelligent systems, suggesting that flexibility is key to overcoming current limitations in robotics.
"for us to you know to optimize our efficiency our evolutionary fitness and so on is to utilize all that experience to build up the best iceberg we can get and that's actually one you know well that so..."
The Iceberg of Knowledge in Learning
In this segment, Sergey delves into the concept of the 'iceberg of knowledge' that humans possess, which is built over a lifetime of experiences. He draws parallels to machine learning, discussing the challenges of distilling unstructured experiences into a common-sense understanding of the world.
"the table everything is fine far knock it over which I'm not going to do but if I were to do that what would happen and I know that nothing good would happen from that but if I have a bad understandin..."
Exploration and Learning in AI
Sergey emphasizes the importance of exploration in AI development, suggesting that intelligent systems should not only maximize utility but also engage in diverse experiences. He proposes a model where exploration prepares systems for uncertain future tasks, enhancing their adaptability.
"problem of maximizing utility like any kind of rational AI agent and then anything you do is in service to maximizing that utility but a very interesting kind of way to look at I'm not necessary sayin..."
The Future of Robotics: Goals and Challenges
In this concluding segment, Sergey reflects on the overarching goals of robotics. He discusses the pragmatic challenges of maximizing the usefulness of robots and the philosophical questions surrounding the future of intelligent systems in a rapidly evolving technological landscape.
"you know the things that they draw inspiration from are the potential for robots to like help us learn about intelligence and about ourselves that's that's fascinating that robotics is basically the s..."
Integrating Perception and Control
Levine explains the challenges of integrating perception and control in robotics, arguing that treating these components together can lead to better performance. He shares insights from his research on end-to-end reinforcement learning for robotic manipulation, demonstrating how combining these elements can optimize task execution.
"might find some new insight so that that could be that could be in any space it doesn't have to be robotics but you're saying yeah I get it's kind of interesting that robotics seems to have a lot of t..."
Learning from Human Heuristics
This segment highlights the importance of learning from human heuristics in robotics. Levine discusses the gaze heuristic, a strategy used by humans and animals to intercept moving objects, and how such insights can inform robotic design and improve performance in dynamic environments.
"hardest problem here or is there or is what you said true that when you start to look at all of them together that's an int that's a whole nother thing like you can't even say which one individually i..."
The Canonical Problem of Robotic Grasping
Levine identifies robotic grasping as a canonical problem in robotics, discussing its complexities and the advancements made in this area. He emphasizes the shift from traditional geometric approaches to learning-based methods that leverage simulation and real-world trial-and-error to enhance robotic grasping capabilities.
"and we you know people who have studied sort of perceptual heuristics in humans and animals find things like that all the time so one one very well-known example this is something called the gaze heur..."
Common Sense in Robotics
Levine explores the necessity of common sense reasoning in robotics, suggesting that understanding robotics can enhance our AI systems' common sense. He argues that common sense emerges from interacting with the world, and current AI systems often lack this due to their limited operational environments.
"that what works really well for robotic grasping instantiated in many different recent works including our own but also ones from many other labs is to use learning methods with some combination of ei..."
Robotics as a Study of Intelligence
In a thought-provoking shift, Levine posits that studying robotics may provide insights into artificial intelligence rather than the other way around. He discusses how robotics can help us understand fundamental learning mechanisms necessary for general intelligence, framing robotics as a key area for exploring AI.
"very clear notion of you did it or you didn't do it so in terms of spilling things there creeps in this notion that starts the sound and feel like common sense reasoning do you think solving the gener..."
Learning vs. Human Expertise in Robotics
Levine addresses the debate on whether robotics can be solved purely through learning methods without human expertise. He argues that while human input is essential in the initial stages, automated optimization techniques can increasingly solve problems that once required meticulous manual engineering.
"essentially maximize their utility whereas the systems we're building now don't have to do that they can take some shortcut that's fascinating you've a couple of times already sort of reframed the rol..."
The Evolution of Control Systems
Levine reflects on the evolution of control systems from classical optimal control to modern machine learning approaches. He highlights how the integration of learning-based systems allows for continuous improvement and adaptation, contrasting it with traditional methods that relied on fixed equations.
"thing I will say on this topic is I don't think this is actually a very radical or very new idea I think people have have been thinking about automated optimization techniques as a way to do control f..."
The Role of Symbolic AI
In this segment, Levine discusses the relevance of symbolic AI in today's machine learning landscape. He explains how the principles of symbolic reasoning have evolved into probabilistic systems and how these concepts still underpin modern AI, despite the shift towards neural networks.
"term in terms of logic you have some query like what action do I take in order to for X to be true and then you manipulate your logical symbolic representation to get an answer what that turned into s..."
Explainability and Storytelling in AI
Levine delves into the importance of explainability in AI systems, linking it to the human desire for storytelling. He discusses how the ability to convey reasoning and decisions is crucial for trust in AI, and how this aspect may differ between traditional expert systems and modern neural networks.
"that there's a human desire for intelligence systems to be able to convey in a poetic way to us why made the decisions it did like tell a convincing story and perhaps that's like a silly human thing l..."
Understanding Reinforcement Learning
Levine provides a comprehensive overview of reinforcement learning, describing it as a modern approach to learning-based control. He explains the concept of learning from rewards and punishments, and how this framework has evolved to encompass broader applications in robotics and AI.
"explainable when errors occur it's just that for other intelligence systems to be in our world we seem to want to tell each other stories and that that's true in the political world is true in the aca..."
Understanding Reinforcement Learning
Levine provides a comprehensive overview of reinforcement learning, defining it as a modern incarnation of learning-based control. He explains how reinforcement learning involves learning from rewards and punishments, and how it can be applied to various decision-making problems, including robotics and AI. This segment clarifies the fundamental principles of reinforcement learning and its broader implications.
"world-class researchers in reinforcement learning deeper and forceful learning certainly in the robotic space what is reinforcement learning i think that reinforcement learning refers to today is real..."
Reinforcement Learning vs. Supervised Learning
In this segment, Levine contrasts reinforcement learning with supervised learning, explaining how reinforcement learning generalizes the latter by relaxing certain assumptions. He discusses the practical applications of both methods and the current limitations of reinforcement learning in real-world scenarios, emphasizing the need for effective algorithms to bridge the gap between the two approaches.
"the applicability of reinforcement learning yeah so rational decision-making is essentially the the encapsulation of the AI problems you didn't through a particular lens so any problem that we would w..."
Challenges of Off-Policy Reinforcement Learning
Levine addresses the challenges faced in off-policy reinforcement learning, particularly the difficulty of utilizing large amounts of prior data effectively. He discusses the importance of developing algorithms that can bootstrap from existing datasets to improve decision-making processes in AI systems, highlighting ongoing research in this area.
"approaches so that so that I my question comes from more practical sense like what do you see is the gap between the more general reinforcement learning and the very specific yes it's a question decis..."
What-If Questions in Reinforcement Learning
This segment delves into the concept of 'what-if' questions in reinforcement learning, explaining how different methods approach these inquiries. Levine outlines the distinctions between model-based and value-based methods, emphasizing the challenges of predicting outcomes for actions that have not been previously taken, which is a core difficulty in reinforcement learning.
"applications of these technologies so this is what's referred to as off policy reinforcement learning or offline RL or batch RL and I think we're seeing a lot of research right now that that's bringin..."
Understanding Policies in Reinforcement Learning
Levine explains the concept of policies in reinforcement learning, describing how they map observations of the world to actions. He discusses the differences between on-policy and off-policy data, and how these concepts impact the effectiveness of reinforcement learning algorithms in various applications, including robotics.
"what-if questions now unfortunately for us with current machine learning methods answering what-if questions can be really hard because they are really questions about things that didn't happen if you..."
Trustworthiness in Off-Policy Learning
In this segment, Levine discusses the importance of trustworthiness in off-policy reinforcement learning. He explains how to assess whether a given action will yield reliable predictions and the role of distribution estimation methods in this process. This highlights the complexities involved in ensuring accurate decision-making in AI systems.
"obviously on policy data is more useful to you because if your current policy makes some bad decisions you will I you see that those decisions are bad off policy data however might be much easier to o..."
The Future of Off-Policy Reinforcement Learning
Levine shares insights on the future of off-policy reinforcement learning, emphasizing that the current challenges are primarily algorithmic. He discusses the potential for breakthroughs through innovative algorithms and the relationship between reinforcement learning and causal inference, suggesting that advancements in these areas could lead to significant progress.
"collect big benchmark data sets that allow us to explore the space is it a new kinds of methodologies like what's your sense or maybe coming together in a space of robotics and defining the problem to..."
The Beauty of Reinforcement Learning
In this reflective segment, Levine articulates the beauty of reinforcement learning, particularly the idea that optimal control can be achieved without a complete model of the world. He discusses the elegance of this approach from a control perspective, highlighting its significance in the broader context of AI and machine learning.
"forward is this to you the most exciting space of reinforcement learning now or is there what's uh and maybe taking a step back not just now but what's to use the most beautiful idea apologize for the..."
Deep Reinforcement Learning Explained
Levine defines deep reinforcement learning as the combination of reinforcement learning algorithms with high-capacity neural networks. He elaborates on the significance of this integration, particularly in overcoming challenges related to feature representation in complex environments, and how it allows for learning directly from raw inputs.
"seems kind of kind of you know very elegant not something that sort of becomes immediately obvious at least in the mathematical sense does it make sense to you that it works at all well I think it mak..."
Challenges in Real-World Reinforcement Learning
Levine addresses the practical challenges of applying reinforcement learning in real-world scenarios, such as the risk of damaging objects during trial-and-error learning. He contrasts human common sense with current robotic learning capabilities, discussing the need for better scaffolding and reward functions to enhance learning efficiency.
"and and people tried all sorts of things that would write down you know an expert chess player looks for whether the the knight is in the middle of the board or not so that's a feature is night in mid..."
Data Efficiency in Reinforcement Learning
In this segment, Levine explores the concept of data efficiency in reinforcement learning, emphasizing the importance of reusing data to avoid excessive trial-and-error learning. He discusses multitask learning and meta-learning as strategies to leverage past experiences for more efficient learning in new tasks.
"other problems like one problem you run into very quickly it'll first sound like a very pragmatic problem that actually turns out to be a pretty deep scientific problem take the robot put in your kitc..."
Tesla Autopilot and Data Utilization
Levine examines Tesla's Autopilot system, discussing how it utilizes data from both human drivers and its own computer vision algorithms. He raises questions about the reliability of the system and the importance of understanding when to trust its predictions, drawing parallels to off-policy reinforcement learning.
"mentioned that's really nice of what how do we push forward there do you think there's there's this kind of sample efficiency question that people bring up or you know not having to break a hundred th..."
The Role of Simulation in Learning
Levine discusses the critical role of simulation in reinforcement learning, noting its practicality for current breakthroughs. However, he argues that long-term success will require machines to learn from real-world data, as reliance on simulations can create bottlenecks that hinder progress.
"that data so I think that actually the kind of problems that come up when we want systems that are reliable and that can kind of understand the limits of their capabilities they're actually very simil..."
Simulation Hypothesis and AI
In a thought-provoking discussion, Levine contemplates the simulation hypothesis, considering the implications of creating realistic simulations for AI training. He reflects on the challenges of simulating human interactions and the complexities they introduce to reinforcement learning problems.
"utilize real experience and this is by the way this is something that I think is quite relevant now especially in the context of some of the things we've discussed because some of these kind of scaffo..."
Interacting with Humans in Robotics
Levine concludes by addressing the complexities of robotic systems interacting with humans. He expresses hope that advanced robotic learning systems can effectively learn from their experiences with people, while also acknowledging the ongoing research needed to improve these interactions.
"seems like put this way it seems kind of weird the aspect of the simulation most interesting to me is the simulation of other humans that seems to be a complexity that makes the robotics problem harde..."
Can We Learn Gravity from Data?
In this segment, Levine tackles the question of whether fundamental concepts like gravity can be learned from data alone. He discusses the pitfalls of assuming prior knowledge is always beneficial and suggests that many important phenomena can be learned through experience rather than explicit instruction.
"task maybe that'll be enough now of course there if it's not enough there are many other things we can do and there's quite a bit of research on that in that area but I think it's worth a shot to see ..."
The Role of Prior Knowledge in Learning
Levine critiques the assumption that providing machines with prior knowledge is always advantageous. He argues that many concepts, such as gravity, can be learned effectively through experience, highlighting the potential for machines to discover important principles autonomously.
"so a very simple clean way to ask that is do you really think we can learn gravity from just data the idea the the laws of gravity so it says something that I think is a common kind of pitfall when th..."
Local Minima in Learning Theories
Levine discusses the challenges of discovering accurate theories in learning, drawing parallels to historical misconceptions in biology and medicine. He emphasizes the importance of developing robust theories that can adapt to new information and avoid getting stuck in local minima.
"space of many local local minima in terms of theories of this world that we would discover and get stuck on yeah of course Newtonian mechanics is not necessarily easy to come by yeah and well in fact ..."
Self-Play in Reinforcement Learning
This segment explores the concept of self-play in reinforcement learning, where agents compete against each other to improve their skills. Levine discusses the potential of this mechanism to enhance learning and its applicability to real-world scenarios, emphasizing the need for machines to interact with their environments.
"of self play is fascinating reinforcement learning sort of these competitive and creating a competitive context in which agents can play against each other in a sort of at the same skill level and the..."
Developing Reward Functions
Levine addresses the complexities of creating effective reward functions in reinforcement learning. He suggests that understanding how to communicate objectives to machines is crucial and explores the idea of intrinsic motivation as a means to guide learning.
"through natural interaction with the world and once we can do that then they can go out and play with they can play with each other they can play with people they can play with the natural environment..."
Unsupervised Reinforcement Learning
In this segment, Levine discusses the concept of unsupervised reinforcement learning, where machines learn useful skills without explicit tasks. He highlights recent research on using information-theoretic measures to guide learning and the potential for machines to discover stable niches in their environments.
"sense of how we develop a reward for good you know good reward functions yeah I think that's a very complicated and very deep question and you're completely right that classically in reinforcement lea..."
Curiosity as a Reward
Levine explores the idea of curiosity as a potential reward mechanism in reinforcement learning. He speculates on whether discovering new things can emerge as a natural consequence of optimizing for capability, indicating ongoing research in this area.
"something that has sometimes been called unsupervised reinforcement learning which i think is a really fascinating area of research especially today we've done a bit of work on that recently one of th..."
Anticipating Unintended Consequences
Levine discusses the challenges of anticipating unintended consequences in reinforcement learning systems. He emphasizes the importance of defining objectives that prevent undesirable behaviors and the need for careful consideration in safety-critical applications.
"domain so we're exploring that pretty actively is there a role for a human notion of curiosity in itself being the reward sort of discovering new things about the war the world so one of the things th..."
Aligning AI with Human Values
In this segment, Levine addresses the alignment of AI systems with human values and interests. He expresses concerns about unintended consequences arising from poorly optimized objectives and emphasizes the need for responsible technology development.
"capability is is there ways to understand or anticipate unexpected unintended consequences of particular reward functions sort of anticipate the kind of strategies that might be developed and try to a..."
Existential Threats of AI
Levine reflects on the existential threats posed by AI systems, highlighting the risks associated with nefarious human intent and the potential flaws in AI systems. He advocates for building responsible technology to mitigate these risks.
"beings do you have thoughts on this they have kind of concerns of where reinforcement learning fits into this or are you really focused on the current moment of us being quite far away and trying to s..."
Learning from AI Systems
Levine discusses the insights that reinforcement learning systems can provide about human behavior and ethical flaws in data. He anticipates that widespread deployment of these systems will reveal interesting behaviors and interactions with humans.
"things like that is more with systems that like need to work better that the optimize their objective better you have thoughts concerns about existential threats of human level intelligence sort of if..."
Transparency in AI Algorithms
In this concluding segment, Levine emphasizes the need for transparency in AI algorithms, particularly in large companies dealing with data. He advocates for understanding the behavior of AI systems and ensuring accountability in their deployment.
"that technology do you think RL systems has something to teach us humans you said nefarious humans getting us in trouble I mean machine learning system self in some ways have revealed to us the ethica..."
Rich Sutton's Bitter Lesson
Levine reflects on Rich Sutton's 'Bitter Lesson' in AI research, which suggests that simple, general methods leveraging computation often yield the best results. He argues for the importance of developing general algorithms that can autonomously collect real-world data, highlighting the challenges of data acquisition in real environments.
"your sense I don't know if you looked at the blog post bitter lesson by Irish Sutton where it looks at serve the big lesson of research in AI in reinforcement learning is that simple methods general m..."
Learning from Human Experience
In this segment, Levine explores the idea of leveraging the collective experiences of humanity to improve machine learning. He discusses the challenges of data collection in the real world and suggests that machines should learn from the vast experiences of people, drawing parallels to how humans learn.
"sense that we should build general methods and we should build the kind of methods that we can deploy and get them to go out there and like collect their experience autonomously I think that you know ..."
Books That Inspire Innovation
Levine shares how Isaac Asimov's works inspired him during his youth, reflecting on the influence of science fiction on his perception of AI and robotics. He discusses the importance of envisioning a future where AI plays a significant role in society and encourages others to explore literature that sparks their imagination.
"through your life would book or books technical or fiction or philosophical had a big impact onion on the way you saw the world I know he thought about in the world your life in general hmm and maybe ..."
The Awakening to AI's Potential
Levine recounts his journey to realizing the potential of artificial intelligence during graduate school. He reflects on a pivotal seminar that shifted his perspective, leading him to believe in the imminent advancements in AI and the significance of being part of this transformative field.
"did you first yourself like fall in love with the idea of artificial intelligence get captivated by this field so my honest answer here is actually that I only really started to think think about it a..."
Advice for Aspiring AI Researchers
In this segment, Levine offers valuable advice for students interested in AI and machine learning. He emphasizes the importance of envisioning desired outcomes and backtracking to identify necessary steps, encouraging a focus on meaningful problems rather than just performance metrics.
"when someone who had been working on that kind of stuff their whole career suddenly says that yeah like that had that had some effect on me yeah this might be a special moment in the history of the fi..."
The Meaning of Life and AI's Role
Levine contemplates the meaning of life and the fulfillment derived from working on impactful problems. He shares his vision of creating machines that continuously improve and adapt, aiming to push the boundaries of complexity in the universe, and concludes with a reflection on the significance of ambition in intelligence.
"think that thinking about that and then backtracking from there and imagining the steps needed to get there will actually do much better research it'll lead to rethinking the assumptions it'll lead to..."
Closing Thoughts and Inspirations
In the closing segment, Levine expresses gratitude for the conversation and reflects on the journey of research in AI and reinforcement learning. He encourages listeners to support the podcast and leaves them with a thought-provoking quote from Salvador Dali about the relationship between intelligence and ambition.
"universe yes well I don't think there's a better way to end it Sergey thank you so much is a huge honor I can't wait to see the amazing work they have to publish and in education space in terms of rei..."