All Segments
LLMs will hit the data wall if they can’t generalize – OpenAI cofounder John Schulman
3 segments available
Full Episode: https://youtu.be/Wo95ob_s_NI Apple Podcasts: https://podcasts.apple.com/us/podcast/john-schulman-openai-cofounder-reasoning-rlhf-plan/id1516093381?i=1000655679622 Spotify: https://open.spotify.com/episode/1ivzHH9RWciXe4O1rKtldf?si=53503781e05f4d8f Transcript: https://www.dwarkeshpatel.com/p/john-schulman/ Me on Twitter: https://twitter.com/dwarkesh_sp/
Segments Timeline
Hitting the Data Wall
John Schulman discusses the potential plateau in the development of language models, suggesting that without significant advancements in generalization, models may hit a 'data wall.' He emphasizes the challenges posed by limited data and the need for evolving pre-training methods as we approach this limit.
"so because there doesn't seem to be a model released since g54 that seems to be significantly better there's seems to be the hypothesis that potentially we're hitting some sort of plateau and that the..."
The Challenge of Generalization
In this segment, Schulman explores the complexities of generalization in AI models, particularly regarding the transfer of learning from different modalities like code and language. He highlights the difficulties in conducting scientific studies on this topic and expresses interest in understanding the results of various data blends on model performance.
"of um pre-training to somewhat change over time as we get closer to it I I think we've talked about some examples generically about generalization one example I was thinking of was the idea that there..."
Domain Expertise in Training
Schulman addresses the necessity of domain expertise in training AI models, questioning whether extensive labeling is required for effective learning. He argues that generalization from a well-trained base model can yield significant results, even without specific examples, suggesting that models can adapt to programming tasks with minimal direct training.
"and see what you get uh so I'm not like um aware of any results uh public like public results on um like ablations um involving code data and reasoning performance and so forth so that would be I woul..."