All Segments
Can synthetic data unlock AI recursive self-improvement? — Mark Zuckerberg
3 segments available
Full Episode: https://youtu.be/bc6uFV9CJGg Apple Podcasts: https://podcasts.apple.com/us/podcast/mark-zuckerberg-llama-3-open-sourcing-%2410b-models-caeser/id1516093381?i=1000652877239 Spotify: https://open.spotify.com/episode/6Lbsk4HtQZfkJ4dZjh7E7k?si=GOqj7hUdSaWSgi7ULWXjMA Transcript: https://www.dwarkeshpatel.com/p/mark-zuckerberg Me on Twitter: https://twitter.com/dwarkesh_sp
Segments Timeline
The Power of Data in AI Training
Mark Zuckerberg discusses the surprising results from training a 70 billion parameter AI model on 15 trillion tokens. He reflects on the ongoing learning capabilities of the model and the balance between further training and moving on to new projects like Llama 4. This segment highlights the potential for future AI training to involve generating synthetic data, emphasizing the evolving nature of AI development.
"one of the interesting things about it we saw even with the 70 billion is we we thought it would get more saturated you know it's like we train it on around 15 trillion tokens we I guess our predictio..."
Synthetic Data and Recursive Improvement
In this segment, Zuckerberg explores the concept of using synthetic data for AI models and its implications for recursive self-improvement. He raises questions about the potential for models like Llama 3 and Llama 4 to become significantly smarter through iterative training processes. The discussion touches on the limitations of current models and the physical constraints that may affect their development.
"more along the lines of inference generating synthetic data to then go feed into the model so I don't know what that ratio is going to be but I I consider um the generation of synthetic data to be mor..."
Balancing AI Development and Global Dynamics
Zuckerberg addresses the importance of maintaining a balance of power in AI development. He discusses the risks of open-sourcing AI architectures and the potential for geopolitical implications, particularly concerning competition between the U.S. and China. This segment emphasizes the need for caution and strategic thinking in the rapidly evolving landscape of artificial intelligence.
"forth um well I think it could within the parameter of whatever the model architecture is it's just that like at some level I don't know I I think like today's 8 billion parameter models I just don't ..."