All Segments
Google DeepMind Lead Researchers on Genie 3 & the Future of World-Building
30 segments available
Genie 3 can generate fully interactive, persistent worlds from just text, in real time. In this episode, Google DeepMind’s Jack Parker-Holder (Research Scientist) and Shlomi Fruchter (Research Director) join Anjney Midha, Marco Mascorro, and Justine Moore of a16z, with host Erik Torenberg, to discuss how they built it, the breakthrough “special memory” feature, and the future of AI-powered gaming, robotics, and world models. They share: - How Genie 3 generates interactive environments in real time - Why its “special memory” feature is such a breakthrough - The evolution of generative models and emergent behaviors - Instruction following, text adherence, and model comparisons - Potential applications in gaming, robotics, simulation, and more - What’s next: Genie 4, Genie 5, and the future of world models This conversation offers a first-hand look at one of the most advanced world models ever created. Timecodes: 0:00 Introduction 0:29 The Evolution of Generative Models 1:10 Real-Time Interactivity & User Experience 4:35 Applications and Use Cases 8:15 The Importance of Special Memory 13:12 Emergent Behaviors & Model Capabilities 19:45 Instruction Following & Text Adherence 20:48 Comparing Genie 3 and Other Models 21:56 The Future of World Models & Modalities 32:23 Robotics, Simulation, and Real-World Impact 37:58 Looking Ahead: Genie 4, 5, and Future World Models 40:41 Are We Living in a Simulation? Resources: Find Shlomi on X: https://x.com/shlomifruchter Find Jack on X: https://x.com/jparkerholder Find Anjney on X: https://x.com/anjneymidha Find Justine on X: https://x.com/venturetwins Find Marco on X: https://x.com/Mascobot Stay Updated: Let us know what you think: https://ratethispodcast.com/a16z Find a16z on Twitter: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Subscribe on your favorite podcast app: https://a16z.simplecast.com/ Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details, please see a16z.com/disclosures.
Segments Timeline
The Magic of Real-Time World Generation
In this segment, the researchers discuss the groundbreaking ability of Genie 3 to generate interactive worlds from minimal input. They express excitement about how this technology can create environments that appear real to users, highlighting the significance of real-time interactivity and the potential for immersive experiences.
"All of the applications basically stem from the ability to generate a world that that just just from a few words. You look at it and like there's a world, you know, that's generated in front of your e..."
Reflections on Genie 3's Impact
Jack Parker-Holder and Shlomi Fruchter reflect on the unexpected response to Genie 3. They share insights into the development process, the challenges faced, and the excitement surrounding the model's capabilities, emphasizing the importance of real-time generation and user engagement.
">> Yeah >> team why don't we reflect internally a little bit about what we found so gamechanging about G3 and why we're so excited to have this conversation Mark. >> Yeah for sure. I mean uh first of ..."
Combining Innovations for Genie 3
The researchers delve into the collaborative efforts that led to Genie 3's creation, discussing the integration of various projects and technologies. They highlight the ambitious goals set during development and the surprising timeline of achieving these innovations.
"right in separate efforts right so we had this G2 project that was much more sort of like 3D environments that it could generate and it wasn't super high quality. It felt like it coming from Genie 1, ..."
The Real-Time Experience: A Game Changer
In this segment, the team emphasizes the importance of real-time interaction in Genie 3. They share personal experiences of testing the model and the 'aha' moments that came with witnessing its capabilities, underscoring the magic of immediate responsiveness in AI-generated environments.
"probably the bit that surprised many of us because obviously we set ourselves these goals and like we tried very hard to achieve them but you can never be totally sure how it's going to actually um fe..."
Applications of Genie 3: Beyond Gaming
The conversation shifts to the diverse applications of Genie 3, from gaming to robotics and education. The researchers discuss the potential for creating personalized experiences and training environments, emphasizing that the future of development will depend on how others leverage this technology.
">> I think you guys like I don't know if this was on purpose or not, but you perfectly timed it uh when everyone on like X and Reddit and everywhere was making those videos of like characters walking ..."
Unlocking Unlimited Environments
Jack Parker-Holder shares insights from his background in reinforcement learning and how it influenced the development of Genie 3. He discusses the vision of creating unlimited environments and the unexpected breadth of applications that emerged during the project.
"from the ability to generate a world that that just just from a few words and I I think uh for me kind like this this potential when I started looking at video models I think it was pretty early when ..."
The Emergence of Special Memory
The researchers introduce the concept of 'special memory' in Genie 3, explaining its significance in maintaining persistence within generated environments. They recount their surprise at how effectively this feature worked, marking it as a major breakthrough in the model's capabilities.
"journey to get there right which is like I personally myself worked in reinforcement learning for a few years before starting the GE project um in 2022. Uh and the motivation originally was like that ..."
The Journey to Real-Time Video Generation
In this segment, the team discusses the evolution of video generation technology and how Genie 3 achieved real-time capabilities. They reflect on the challenges faced and the excitement of reaching a point where users can interact with generated content seamlessly.
"So I think it's like language models in 2021 maybe you probably wouldn't have guessed like an IMO gold medal a few years later would would come that fast. Um but as a direct application of that techno..."
From Genie 2 to Genie 3: A Leap in Memory
The conversation shifts to the evolution from Genie 2 to Genie 3, highlighting the advancements in memory capabilities. Jack elaborates on how Genie 2 had limited memory, while Genie 3 aimed for a more ambitious goal of over a minute of memory, achieving higher resolution and real-time performance.
"about um when did you discover that as an emergent property or was that a specific design goal? What's the backstory on that because that feels like a big unlock. Jack, uh why don't we start with you?..."
Emergent Properties and Technical Challenges
Jack and Shlomi delve into the technical challenges faced during the development of Genie 3. They discuss the emergent properties of the model, emphasizing the importance of generating frame-by-frame interactions without relying on explicit representations, which enhances the model's generalization capabilities.
"those are kind of conflicting objectives, right? Um, so we set ourselves this kind of technical challenge. Uh, and we said like if we target this then it's just about feasible and it'll be pretty incr..."
Real-Time Interaction: The Magic of Consistency
The segment explores the excitement of users experiencing Genie 3's real-time interactions for the first time. Jack shares his observations on how users react when they see the model maintain consistency in its generated environments, reinforcing the model's impressive capabilities.
">> Every time someone interacts with it for the first time and they like test they look away and then look back I'm always like holding my breath and then and then it looks back and it's the same. I'm..."
Scaling Up: Unexpected Behaviors
Shlomi discusses the scaling of generative models and the unexpected behaviors that emerge as data and compute resources increase. He highlights how Genie 3 exhibits improved understanding of interactions, such as agents responding appropriately to environmental cues, showcasing the model's advanced capabilities.
"obviously eventually you'd want to >> one more question related on the between Genie one to like you know like in for example NLMs like you have like DeepCar one like they saw in this paper like the l..."
Realism in Simulations: A Leap Forward
The researchers reflect on the significant improvements in realism from Genie 2 to Genie 3, particularly in physics simulations and environmental interactions. They share examples of breathtaking water simulations and how the model's output can be perceived as photorealistic by non-experts.
"trends that we've we've observed. >> Yeah. And from G2 to three it's like I think the real world capabilities really increased. Right. So on the physics side, um some of the water um simulations you c..."
Understanding Interactions in Diverse Environments
Jack and Shlomi explain how Genie 3's training allows it to understand character interactions across various terrains and environments. They discuss the emergent properties of the model that enable it to generate realistic behaviors, such as swimming in water or skiing downhill.
"in that side. >> Yeah. One of the things that was really cool in all the examples was the water is sort of a great way to see like does it understand like what the world is and how objects interact an..."
Balancing Consistency and Creativity
The segment addresses the trade-offs in model design between creating consistent worlds and allowing for creative, unexpected interactions. Jack and Shlomi discuss how Genie 3 manages to navigate these challenges, providing users with exciting experiences that go beyond typical expectations.
"sort of interactions should differ given the the like terrain that they're in. And I think that that really is a property of of scale and breadth of of training. So um this is very much like an emerge..."
Navigating Unlikely Scenarios
The conversation shifts to the challenges of generating unlikely scenarios within Genie 3. The researchers explain the tension between creating realistic environments and adhering to user prompts, showcasing the model's surprising success in navigating low-probability situations.
"what we want right like many people they don't want to just look at the video that looks like their their own no maybe this room um uh but but more something a bit more exciting and that's where like ..."
Text-Driven World Creation
Jack Parker-Holder elaborates on the advancements in Genie 3's ability to generate worlds directly from text descriptions. He contrasts this with previous models that relied on image prompting, emphasizing the enhanced controllability and creativity that text-based inputs provide.
"pretty amazing. Um, so I think that that's actually a really important capability that we didn't have with Genie2 as well, right? Because we relied on image prompting. And so there was some transfer i..."
The Distinction Between Genie 3 and V3
The researchers discuss the differences between Genie 3 and V3, highlighting Genie 3's interactive capabilities and the absence of audio in V3. They explain the rationale behind naming it Genie 3, focusing on its unique features and the ongoing research nature of V3.
"areas that we can like seek out advice and help from. And Shomi, a question for you on that is, you know, having led the V3 work, which is kind of mind-blowing, is is there a reason why this is Genie ..."
Exploring Modalities and Future Directions
The discussion delves into the boundaries of different modalities in AI, including video generation and real-time world modeling. The researchers speculate on whether these fields will converge or diverge in the future, emphasizing the importance of control and speed in creating new samples.
"Um it seems like they share kind of one one parent today which is you know video generation but where is the world going do you think are these two completely different fields? From my perspective um ..."
Balancing Research and Practical Applications
The segment concludes with a reflection on the balance between research-driven innovation and practical applications. The researchers acknowledge that while they have potential use cases in mind, the primary focus remains on advancing the technology itself.
">> Yeah, I think this is a really interesting point, mate. And ultimately it has to be driven by like technical decisions um and also like the the goals, right? So we if you look at the the models rig..."
The Future of Genie Models
The team reflects on the future of Genie models, particularly Genie 4 and 5, and the potential applications in gaming and multi-user environments. They discuss the importance of scaling models and the excitement around discovering new applications as more users interact with the technology. The segment captures their vision for the evolution of AI and its impact on various fields.
"talking about it, it sounds like you've also been pretty thoughtful around what are the different capabilities or features needed for different potential use cases at least of different models. Yeah, ..."
Building Capable AI Agents
The researchers share insights on their approach to building capable AI agents and the significance of feedback in model development. They discuss the importance of creating environments that allow agents to learn from experience, drawing parallels to successful models like AlphaGo. This segment emphasizes the need for realistic simulations to enhance the learning process in robotics.
"sure that that uh over time um there is more access to to um to the models we build um and and I think that's the only way to discover what's the real potentials. >> I guess one one somewhere related ..."
Simulating Real-World Scenarios
The conversation shifts to the challenges of simulating real-world scenarios in robotics. The researchers explain the limitations of current robotic simulations and the importance of bridging the gap between simulated and real environments. They highlight the need for AI to navigate complex, everyday situations, emphasizing the potential for Genie 3 to contribute to advancements in robotics.
"applications I never thought of that come up from other people seeing the model. Right. So, I think it's kind of this like trade-off of, you know, obviously you want to focus on some applications, but..."
Genie 3 and Robotics Integration
The segment explores the integration of Genie 3 with robotics, focusing on the potential for creating interactive environments for robotic agents. The researchers discuss the importance of composability in AI systems and how Genie 3 can facilitate real-time interactions between agents and their environments. This highlights the innovative applications of AI in enhancing robotic capabilities.
">> I'm actually personally petrified of skiing and the models are already quite good at that. So, I might when things quieten down, spend some time cuz I promised my my wife that our children would gr..."
Overcoming Robotics Limitations
Jack Parker-Holder discusses the current limitations in robotics and how Genie 3 serves as an environment model rather than an agent. He emphasizes the importance of learning from experience and the challenges of simulating real-world scenarios, highlighting the gap between simulation and reality in robotics.
"today that you think we'd have to overcome as a space to make the robotics um sort of progress the rate of progress in robotics much faster than it is now >> so um we designed it to be an envir enviro..."
The Challenge of Real-World Learning
In this segment, Parker-Holder elaborates on the difficulties of learning from experience in physical environments. He explains the need for robots to navigate complex real-world situations, such as walking a dog, and the limitations of current data-driven approaches in robotics.
"the best ones at deep mind we have Majoko right which we work with um they're still quite far away from the real world right and so you have the sim tore gap um but even the sim tore gap itself uh I t..."
Genie 3: Bridging Simulation and Reality
Parker-Holder highlights how Genie 3 combines the strengths of real-world data-driven approaches with the ability to learn in simulation. He discusses the potential of Genie 3 to revolutionize robotics by enabling robots to learn from simulated experiences effectively.
"not just for for robot example, but I really love this idea of having when it rains in London a lot. Uh, not having to take my dog for the second walk would be great. >> And as you can see, we build a..."
Future Directions in Robotics
The conversation shifts to the future of robotics, with Parker-Holder and Fruchter discussing the importance of reasoning about environments. They explore how Genie 3 can help bridge gaps in physical understanding and improve robotic responses to real-world stimuli.
">> I mean, I personally love California, but my wife's not my wife's not convinced. Sorry. >> We're convinced here. >> Yeah. Just just to touch on, you know, maybe a final point on the robots kind lik..."
The Curve of World Models Development
Fruchter answers a question about the development curve of world models, comparing it to language models. He discusses the current capabilities of Genie 3 and the potential for exponential progress in the field, emphasizing the need for continuous innovation.
"don't know if you can answer this but like is it going to become public like can developers access it at some point or is there like some sort of idea on this? >> So, as you can see, we are very excit..."
Are We Living in a Simulation?
The discussion takes a philosophical turn as the hosts ponder whether we live in a simulation. Parker-Holder shares his thoughts on the nature of reality and the limitations of current hardware, suggesting that quantum computing might play a role in understanding our existence.
"himself right like it's not the case that you're dropping yourself in the world right and like it's like the real being in the real world for example it's actually quite different to that when you do ..."