All Segments
Richard Sutton – Father of RL thinks LLMs are a dead end
40 segments available
Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end. After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need *some* new architecture to enable continual learning. And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals. This new paradigm will render our current approach with LLMs obsolete. In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew. A big thanks to the Alberta Machine Intelligence Institute for inviting me up to Edmonton and for letting me use their studio and equipment. Enjoy! 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒 * Transcript: https://www.dwarkesh.com/p/richard-sutton * Apple Podcasts: https://podcasts.apple.com/us/podcast/richard-sutton-father-of-rl-thinks-llms-are-a-dead-end/id1516093381?i=1000728584744 * Spotify: https://open.spotify.com/episode/3zAXRCFrHPShU4MuuIx4V5?si=c9f4bf24fb4c43e3 𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 * Labelbox makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at https://labelbox.com/dwarkesh * Gemini Deep Research is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at https://gemini.google.com/ * Hudson River Trading doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at https://hudsonrivertrading.com/dwarkesh To sponsor a future episode, visit https://dwarkesh.com/advertise 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – Are LLMs a dead end? 00:13:51 – Do humans do imitation learning? 00:23:57 – The Era of Experience 00:34:25 – Current architectures generalize poorly out of distribution 00:42:17 – Surprises in the AI field 00:47:28 – Will The Bitter Lesson still apply after AGI? 00:54:35 – Succession to AI
Segments Timeline
Reinforcement Learning vs. LLMs
Richard Sutton discusses the fundamental differences between reinforcement learning (RL) and large language models (LLMs). He argues that while LLMs excel at mimicking human language, they lack the ability to understand and learn from their environment, which is essential for true intelligence. Sutton emphasizes that RL focuses on understanding the world and learning from experience, contrasting it with LLMs' reliance on imitation.
"Today I'm chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning and inventor of many of the main techniques used there, like TD learning and policy gradient ..."
The Nature of Learning
Sutton critiques the notion that LLMs possess a world model, asserting that they merely replicate human responses without understanding the underlying reality. He highlights the importance of learning from actual experiences rather than just imitating actions, emphasizing that true learning involves predicting outcomes based on one's actions.
"You would think that to emulate the trillions of tokens in the corpus of Internet text, you would have to build a world model. In fact, these models do seem to have very robust world models. They..."
Imitation Learning vs. Experience
In this segment, Sutton challenges the idea that imitation learning provides a solid foundation for LLMs. He argues that without a clear definition of 'right' actions or goals, LLMs cannot effectively learn or adapt. Sutton insists that reinforcement learning, which includes a goal-oriented approach, is necessary for meaningful learning and adaptation.
"They learn from "here's a situation, and here's what a person did". Implicitly, the suggestion is you should do what the person did. I guess maybe the crux, and I'm curious if you disagree with t..."
The Importance of Goals
Sutton emphasizes that having a goal is essential for intelligence. He argues that LLMs lack substantive goals, which limits their ability to learn and adapt. In contrast, reinforcement learning systems are designed to achieve specific goals, allowing them to define right actions based on rewards and feedback.
"You will say something and you will not get feedback about what the right thing to say is, because there's no definition of what the right thing to say is. There's no goal. If there's no goal, t..."
Predictive Capabilities of LLMs
Sutton discusses the limitations of LLMs in terms of predictive capabilities. He argues that while LLMs can predict the next token in a sequence, they do not possess the ability to adapt or learn from unexpected outcomes. This lack of adaptability is a significant drawback compared to reinforcement learning systems that learn from real-world interactions.
"a prediction about what will happen next. If you say something in your conversation, the large language models have no prediction about what the person will say in response to that or what the re..."
Math Problem Solving and Goals
In this segment, Sutton addresses the success of LLMs in solving math problems, questioning whether this indicates they have meaningful goals. He distinguishes between computational tasks and real-world learning, asserting that true understanding of the physical world requires experiential learning, which LLMs currently lack.
"Isn't that literally what next token prediction is? Prediction about what's next and then updating on the surprise? The next token is what they should say, what the actions should be. It's not w..."
The Bitter Lesson and LLMs
Sutton reflects on his influential essay 'The Bitter Lesson' and its implications for LLMs. He argues that while LLMs utilize vast amounts of computation, they are still reliant on human knowledge, which may limit their scalability. Sutton predicts that future systems capable of learning from experience will surpass LLMs, illustrating the ongoing relevance of his 'bitter lesson' thesis.
"The math problems are different. Making a model of the physical world and carrying out the consequences of mathematical assumptions or operations, those are very different things. The empirical w..."
The Need for New Architectures
Sutton argues for the necessity of developing new architectures for experiential learning, rather than relying on LLMs as a foundation. He emphasizes that while starting with human knowledge may seem beneficial, it often leads to limitations and a lack of adaptability in learning systems.
"I guess that doesn't seem like the crux to me. I think those people would also agree that the overwhelming amount of compute in the future will come from learning from experience. They just think ..."
Imitation Learning in Humans
Sutton and his interviewer discuss the role of imitation learning in human development. Sutton challenges the idea that imitation is the primary learning mechanism for infants, arguing instead that humans learn through trial and error and experience, which aligns more closely with reinforcement learning principles.
"Large language models are trying to get by without having a goal or a sense of better or worse. That's just exactly starting in the wrong place. Maybe it's interesting to compare this to humans. In..."
Understanding Animal Learning
Sutton emphasizes the importance of understanding animal learning processes to replicate intelligence in AI. He argues that supervised learning is not a natural part of animal learning, and that insights from animal behavior can inform the development of more effective AI systems that learn from experience rather than imitation.
"hands around and moving their eyes around. There's no imitation for how they move their eyes around or even the sounds they make. They may want to create the same sounds, but the actions, the thin..."
The Complexity of Human Skills
In this concluding segment, Sutton discusses the complexity of human skills and the long processes involved in mastering them. He highlights the need for AI systems to learn from experience over time, drawing parallels between human learning and the development of intelligent systems.
"We don't have examples of desired behavior. What we have are examples of things that happen, one thing that followed another. We have examples of, "We did something and there were consequences." B..."
Continual Learning in Mammals vs. AI
Sutton contrasts the continual learning capabilities of mammals with the limitations of current AI systems. He points out that while all mammals can learn continuously from their environments, AI lacks this ability, highlighting a significant gap in AI development. This segment underscores the need for AI to adopt more natural learning processes to achieve human-like intelligence.
"I like the way you consider that obvious, because I consider the opposite obvious. We have to understand how we are animals. If we understood a squirrel, I think we'd be almost all the way there t..."
The Experiential Learning Paradigm
Sutton introduces the concept of the experiential learning paradigm, which focuses on the continuous cycle of experience, action, and reward. He explains that intelligence should be centered around altering actions to maximize rewards based on experiences, emphasizing the importance of learning from the stream of experiences rather than relying on pre-defined training phases.
"reason through how to, let’s say, hunt a seal if you're living in the Arctic. There's this many, many-step, long process of how to make the bait and how to find the seal, and then how to process ..."
Reward Functions and Learning
In this segment, Sutton discusses the nature of reward functions in AI and how they can vary depending on the context. He explains that while rewards can be specific to tasks like winning a game or gathering food, a more intrinsic motivation related to understanding one's environment is crucial for effective learning. This highlights the complexity of designing AI systems that can learn and adapt in real-time.
"It's what distinguishes us, perhaps, from many animals. But we're an animal first. We were an animal before we had language and all those other things. I do think you make a very interesting poi..."
Redefining AI Learning Models
Sutton critiques the traditional notion of AI models, suggesting that a better term would be 'the network' to reflect the interconnected nature of learning. He emphasizes the need for AI to learn from experiences and interactions, rather than being confined to static models, which limits their ability to adapt and grow over time.
"This alternative paradigm that you're imagining… The experiential paradigm. Let's lay it out a little bit. It says that experience, action, sensation—well, sensation, action, reward—this happens ..."
Contextual Learning in AI
Sutton explores the concept of contextual learning in AI, arguing that effective learning requires capturing a wide range of information from the environment. He discusses how humans learn from various contexts and experiences, suggesting that AI should similarly integrate diverse information to enhance its learning capabilities.
"When you're imagining this future continual learning agent… They're not "future". Of course, they exist all the time. This is what the reinforcement learning paradigm is, learning from experience..."
Temporal Difference Learning and Long-Term Goals
In this segment, Sutton explains how temporal difference learning can help AI achieve long-term goals by breaking them down into smaller, manageable rewards. He draws parallels between human goal-setting in complex tasks and how AI can learn to navigate similar challenges through incremental learning.
"Do we also get rid of this paradigm where there's the model and then instances of the model or copies of the model that are doing certain things? How do you think about the fact that we'd want th..."
The Role of Context in Human Learning
Sutton discusses the importance of context in human learning, emphasizing that humans acquire knowledge through real-world experiences that cannot be fully anticipated. He argues that AI must similarly learn from its environment to become effective, highlighting the limitations of current models that rely solely on pre-existing knowledge.
"you're going to need this capability. Suppose a human is trying to make a startup. This is a thing which has a reward on the order of 10 years. Once in 10 years you might have an exit where you ge..."
Generalization Challenges in AI
Sutton addresses the challenges of generalization in AI, noting that current models often struggle to transfer knowledge across different tasks or states. He emphasizes the need for improved algorithms that promote effective generalization, rather than relying on human intervention to sculpt learning outcomes.
"You also want some ability for information that you're learning. One of the things that makes humans quite different from these LLMs is that if you're onboarding on a job, you're picking up so m..."
Learning from Experience vs. Reward
In this segment, Sutton contrasts the learning processes of AI with those of humans, arguing that humans learn from a rich tapestry of experiences beyond just rewards. He suggests that AI must develop mechanisms to capture and integrate various forms of information to enhance its learning capabilities.
"It seems to me you need two things. One is some way of converting this long-run goal reward into smaller auxiliary predictive rewards of the future reward, or the future reward that leads to the ..."
The Transition Model of the World
Sutton introduces the concept of the transition model of the world, which encompasses the understanding of cause and effect in learning. He argues that AI must develop a robust model of the world based on experiences, rather than relying solely on rewards, to achieve effective learning and adaptation.
"What is the learning process which helps you capture that information? Now I want to talk about the base common model of the agent with the four parts. We need a policy. The policy says, "In the ..."
Transfer Learning in Reinforcement Learning
Sutton discusses the limitations of current reinforcement learning techniques in achieving transfer learning across different tasks. He emphasizes the need for advancements in AI that allow for better generalization and transfer of knowledge, which is crucial for developing more sophisticated and adaptable AI systems.
"learned from reward. It's learned from, "You did things, you saw what happened, you made that model of the world." That will be learned very richly from all the sensation that you receive, not j..."
The Quest for General Learning Agents
Richard Sutton discusses the limitations of current AI architectures, particularly in the context of AlphaZero, and emphasizes the need for a general learning agent that can transfer knowledge across different states rather than just tasks. He highlights the lack of effective methods for promoting transfer in reinforcement learning and critiques the reliance on human-designed representations for generalization.
"it could be a general learning agent? The idea is totally general. I do use all the time, as my canonical example, the idea of an AI agent is like a person. People, in some sense, have just one w..."
The Pitfalls of Generalization in AI
Sutton elaborates on the challenges of generalization in deep learning models, explaining that while models can solve specific problems, they often fail to generalize effectively to new data. He points out that gradient descent alone does not ensure good generalization, leading to catastrophic interference when new information is introduced.
"generalize well from one state to another state. We don't have any methods that are good at that. What we have are people trying different things and they settle on something, a representation tha..."
LLMs and the Scope of Generalization
In this segment, Sutton contrasts the capabilities of large language models (LLMs) with earlier systems, noting their improved ability to generalize across complex mathematical problems. He questions whether solving problems within a category truly represents generalization, emphasizing the need for a deeper understanding of how LLMs achieve their results.
"The fact that you generalize is not necessarily good or bad. You can generalize poorly, you can generalize well. Generalization always will happen, but we need algorithms that will cause the ge..."
Complexity and Coding Agents
Sutton discusses the evolving capabilities of coding agents, noting their increasing proficiency in generating satisfactory design architectures for developers. He argues that while LLMs can find solutions, they lack inherent mechanisms to ensure effective generalization across various problem-solving methods.
"category as an example of generalization. Let me know if I'm misunderstanding that. Large language models are so complex. We don't really know what information they have had prior. We have to guess..."
Surprises in AI Development
Reflecting on his long career in AI, Sutton shares his thoughts on the surprising effectiveness of large language models and the dominance of simple, general-purpose methods over human-imbued systems. He highlights the gratifying success of reinforcement learning techniques, particularly in applications like AlphaGo and AlphaZero.
"But if there are many ways to solve it, some which generalize well, some which generalize poorly, there's nothing in the algorithms that will cause them to generalize well. But people, of course,..."
The Evolution of AI Techniques
Sutton discusses the historical context of AI breakthroughs, particularly the transition from earlier reinforcement learning methods to modern applications like AlphaGo. He emphasizes that while new applications may seem groundbreaking, they often build on established techniques, showcasing the importance of scaling and innovation in AI.
"I want to zoom out and ask about being in the field of AI for longer than almost anybody who is commentating on it, or working in it now. I'm curious about what the biggest surprises have been. H..."
The Future of AI and Human Intelligence
In this segment, Sutton explores the implications of achieving AGI and the potential for superintelligent systems. He discusses the inevitability of AI succession and the need for a unified understanding of intelligence, while also considering the diverse outcomes that could arise from this evolution.
"old days of AI, what would happen. Learning and search have just won the day. There's a sense in which that was not surprising to me because I was always hoping or rooting for the simple basic pri..."
Designing Intelligence: A New Era
Sutton presents his vision of a future where AI is designed rather than replicated, marking a significant transition in the understanding of intelligence. He argues that as we move towards designed intelligences, we will gain the ability to manipulate and enhance intelligence in ways that were previously unattainable.
"AlphaZero used TD. AlphaZero was applied to all the other games and it did extremely well. I've always been very impressed by the way AlphaZero plays chess because I'm a chess player and it just ..."
The Role of AGI in Future Intelligence
Sutton discusses the potential for AGI to lead to superhuman levels of intelligence and the implications of having many AI researchers contributing to the field. He raises questions about the future of intelligence and the importance of understanding how these systems will evolve and interact.
"a classicist rather than as a contrarian. I go to what the larger community of thinkers about the mind have always thought. Some sort of left-field questions for you if you'll tolerate them. The w..."
The Complexity of AI Collaboration
In this segment, Sutton contemplates the future of AI collaboration and the potential for AIs to spawn copies of themselves for decentralized learning. He discusses the challenges of integrating new knowledge and the risks associated with incorporating external information into an AI's core understanding.
"I guess there are different gradations. I'm not sure your idea makes sense because it seems to presume the existence of AGI and that we've already worked that out. Maybe one way to motivate this i..."
Cybersecurity in the Age of AI
Sutton highlights the emerging concerns of cybersecurity as AI systems become more capable of integrating external information. He warns of the potential dangers of incorporating unverified knowledge and the need for robust safeguards to prevent corruption and maintain the integrity of AI systems.
"other the way cultural evolution works in people? Maybe we should talk about that. The bitter lesson, who cares about that? That's an empirical observation about a particular period in history. 70 ..."
Inevitability of AI Succession
Sutton outlines his four-part argument for the inevitability of AI succession, emphasizing the lack of consensus in governance, the eventual understanding of intelligence, and the rise of superintelligent systems. He discusses the implications of this transition and the potential for both positive and negative outcomes.
"I think this will become a big concern, particularly when you're like, "Oh, he's figured out all about how to play some new game or he's studied Indonesia, and you want to incorporate that into ..."
A New Era of Designed Intelligence
In this concluding segment, Sutton reflects on the transition from replication to design in the evolution of intelligence. He posits that as we develop designed AIs, we will gain a deeper understanding of intelligence itself, allowing for unprecedented advancements in technology and society.
"You have a perspective that's quite different from a lot of people that I've interviewed and a lot of people generally. I also think it's a very interesting perspective. I want to hear about it. I..."
The Ethical Dilemma of AI Offspring
Sutton raises the ethical question of whether designed intelligences should be considered part of humanity. He discusses the implications of viewing AIs as our offspring, emphasizing the importance of how we perceive and interact with these entities as they evolve and gain capabilities.
"I mark this as one of the four great stages of the universe. First there's dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we're giving rise to designed entiti..."
Concerns About Future Generations
This segment delves into the potential concerns surrounding future generations of humans and AIs. Sutton draws parallels between the evolution of species and the rise of more capable intelligences, cautioning against the rapid attainment of power by entities we do not fully understand.
"I do think there are potential worlds with future humans that we would be quite concerned about. Are you thinking like, maybe we are like the Neanderthals that give rise to Homo sapiens. Maybe Hom..."
Navigating Change and Control
Sutton discusses humanity's limited control over the future and the importance of focusing on local goals. He emphasizes the need for a balanced approach to change, advocating for robust values in the development of AI while recognizing the unpredictability of future outcomes.
"Maybe it's the best thing that there has been, but it's far from perfect. I guess there are different varieties of change. The Industrial Revolution was change, the Bolshevik Revolution was also c..."
Teaching Values to AI
In this segment, Sutton explores the idea of instilling values in AI systems. He suggests that while we may not agree on universal morals, we can still impart high integrity and prosocial principles to AIs, ensuring they act in ways that are beneficial to society.
"future has to evolve this way that I want it to." Because then we'll have arguments where different people think the global future should evolve in different ways, and then they have conflict. We..."
The Ongoing Human Enterprise
Sutton concludes by reflecting on the ongoing human enterprise of designing society and values. He notes that despite technological advancements, the fundamental challenges of human existence remain, encapsulating the idea that 'the more things change, the more they stay the same' in the context of AI development.
"So we're trying to design the future and the principles by which it will evolve and come into being. The first thing you're saying is, "Well, we try to teach our children general principles whic..."