All Segments
AI & Evolution: Learning to do More with Less | David Ha | Episode 146
33 segments available
David Ha is the Head of Strategy at Stability AI, and one of the top minds working in AI today. He previously worked as a research scientist in the Brain team at Google. David is particularly interested in evolution and complex systems, and his research explores how intelligence may emerge from limited resource constraints. He joins the show to discuss the advantages of open-source models, modelling AI as an emergent system, why large language models are bad at maths and MUCH more! Important Links: - Teaching Machines to Draw - https://blog.otoro.net/2017/05/19/teaching-machines-to-draw/ (2017) - Weight Agnostic Neural Networks- https://weightagnostic.github.io) (2019 00:00 Start 01:15 Main Podcast 02:36 Why David joined Stability AI 07:08 The advantages of open source models 16:06 We cannot predict the inventions of tomorrow 23:33 Making memes with generative AI 25:41 The centaur approach to AI 29:27 An introduction to large language models 39:28 The relationship between complex systems & resource constraints 47:57 Large language models are bad at maths 58:18 Modelling AI as an emergent system 01:02:22 Understanding different perspectives and much More!
Segments Timeline
The Open Source Ecosystem
David Ha discusses the importance of an open-source foundation for AI technologies. He expresses concerns about a few individuals controlling powerful technologies and emphasizes that a broader ecosystem will benefit society. He highlights the need for models to be accessible on consumer-level hardware, allowing for innovation and understanding of AI systems.
"One of those concerns whenever we have new 📍 technologies is, okay, this is gonna, you know, do harm to the world and only the adults in the room. Make the technology available and be qualified to pu..."
Joining Stability AI
David Ha shares his journey to joining Stability AI, driven by his passion for making machine learning technology broadly available. He reflects on the challenges of scaling AI models and the need for open-source alternatives to closed systems. His decision to join Stability AI was influenced by the desire to contribute to a platform that supports open-source efforts.
"Emergent type models, they are far more adaptive and far more similar to complex adaptive systems. We have like a hundred billion neurons and after a certain number of neurons, these, these phenomena ..."
The Power of Open-Source Models
Ha elaborates on the advantages of open-source models, particularly in the context of generative AI. He contrasts the capabilities of large models with the accessibility of smaller, open-source models like DALL-E Mini and Stable Diffusion, which allow a wider audience to innovate and develop new applications.
"Jim O'Shaughnessy: Yes, absolutely. And we're going to get to that, because that's a mutual interest of both of ours, sort of the collective mind. I call it the human colossus. Now that we have AI, I ..."
Innovation Through Tinkering
David Ha emphasizes the importance of tinkering and innovation in technology. He draws parallels between the evolution of personal computing and the current landscape of AI, suggesting that significant advancements will come from individual developers working with accessible models rather than large corporations.
"on their machines. These are, at the time, they need to be productionalized and served on large corporate machines. So to me, it's a double-edged sword. As you scale up the model, you get better capab..."
The Role of Constraints in AI Development
Ha discusses how constraints can drive innovation in AI. He argues that developing algorithms that perform well with limited resources can lead to breakthroughs, similar to how human intelligence operates efficiently with minimal energy. This philosophy underpins the development of models like Stable Diffusion.
"text. David Ha: So I became totally obsessed with text to image models. And when I decided to leave Google to work on open-source efforts, I focused on open-source models like DALL-E Mini, which went ..."
The Future of Open-Source AI
David Ha shares his vision for the future of open-source AI, highlighting the need for a balance between open and closed systems. He believes that fostering an open-source environment will lead to greater innovation and a more robust technological landscape, benefiting society as a whole.
"And as you said, I could have just started a small company with a few ex Googlers and raised some VC funding, worked on some model to do a product market fit and try to bootstrap from there, get a few..."
Philosophy of Open Collaboration
Jim O'Shaughnessy and David Ha discuss the philosophical underpinnings of open collaboration in technology. They reflect on the historical significance of open-source movements and the importance of allowing diverse minds to contribute to innovation, ensuring that future generations have access to powerful technologies.
"So it is just more fun, in my opinion. Jim O'Shaughnessy: Yeah, I agree. And I am passionate about open-source too. That's why I invested in the company. I'm ruled by a Taoist-Stoic philosophy, which ..."
The Legacy of Open-Source Innovators
Ha and O'Shaughnessy draw inspiration from historical figures like John von Neumann, who advocated for open access to knowledge. They discuss how the hacker culture of tinkering has historically driven innovation and how this spirit is essential for the future of AI development.
"Plus, on top of it, when you look at the history of open-source, it runs the internet now. And in studying open-source versus closed source, I just did a recording earlier today with a guy who wrote a..."
The Importance of Community in AI
David Ha emphasizes the role of community in advancing AI technologies. He argues that a collaborative approach allows for a deeper understanding of AI systems and encourages innovation that would not be possible in closed environments. The discussion highlights the significance of collective efforts in addressing challenges and exploring new possibilities.
"Back in the days when you had the hobbyist tinkering around with building their machines. When I was a kid, people didn't buy a shiny MacBook. You buy a box and then you put your PCI cards in the box,..."
Consumer-Level AI Innovations
Ha discusses how innovations in AI will increasingly occur at the consumer level, similar to the evolution of personal computing. He believes that making AI models accessible on consumer hardware will empower individuals to create and innovate, leading to a new wave of technological advancements.
"use of these models as well. David Ha: Because one of the concerns whenever we have new technologies is, okay, this is going to do harm to the world and only the adults in the room should make the tec..."
Maximizing Efficiency in AI Development
David Ha outlines his strategy for developing AI models that maximize efficiency with minimal resources. He emphasizes the importance of creating algorithms that can perform effectively on consumer-level hardware, aligning with the broader goal of making AI accessible and sustainable.
"David Ha: For example, say something like Stable Diffusion would fit on a consumer level GPU. And because of that, you suddenly have a million people that could develop on it, and you get efforts, you..."
The Future of AI and Human Intelligence
Ha concludes by reflecting on the relationship between AI and human intelligence. He argues that understanding how humans operate with limited resources can inform the development of AI systems that are not only powerful but also efficient and sustainable.
"David Ha: The parallel I was thinking of is, back in the day there were innovations in craze super computers in the mini computer, which is a mega computer and all of these computers. But the real inn..."
The Power of Open Architecture
Jim O'Shaughnessy and David Ha explore the advantages of open architecture in AI, highlighting how cognitive diversity fosters innovation. They argue that allowing a multitude of contributors to enhance AI systems leads to more robust advancements compared to closed systems.
"Jim O'Shaughnessy: Yeah, because constrained resources, and that was one of the things that I also was drawn instantly to the open architecture, because of the idea when you study innovations, when yo..."
The Unknown Innovations of Tomorrow
The conversation shifts to the unpredictability of future innovations. David Ha and Jim O'Shaughnessy reflect on how even the brightest minds of the past could not foresee the internet or quantum physics, stressing the importance of keeping an open mind about future technological advancements.
"Jim O'Shaughnessy: I mean, there's a wonderful quote that I love that is, you can't ask a person to make a list of ideas that would never occur to them. And so I often find when I talk to even very sm..."
Embracing Unknown Unknowns
David Ha introduces the concept of 'unknown unknowns' in technology, emphasizing that the best way to foster innovation is to acknowledge our ignorance about future developments. He cites the importance of creating environments where unexpected discoveries can occur.
"Jim O'Shaughnessy: And so it's this idea that you share with me that, God, no, we don't know everything. And the ability to allow many, many minds of varying interests, to use the base technology, I t..."
The Role of Open-Source Models
Ha discusses the impact of open-source generative models, particularly in creative applications. He shares an example of a father using AI to transform his child's doodles into high-quality images, illustrating the potential for deeper human interaction and creativity through AI.
"simply don't know the list of all of the great things that's going to happen and you want to put technology in a place where these things can be discovered. You want to maximize, to quote my friend Ke..."
AI as a Creative Companion
David Ha reflects on the feedback loop created by AI in artistic endeavors, emphasizing how generative models can enhance creativity and communication between individuals. He envisions a future where children naturally integrate AI into their creative processes.
"David Ha: What I thought was really notable to me is when you pair up these models with creativity in terms of letting kids draw. So I was blown away with the example of a father playing around with t..."
Cultural Commentary through AI Art
Ha discusses the use of generative AI to create culturally relevant art, particularly in response to societal issues. He highlights how open-source models allow for the expression of contemporary messages, showcasing the potential for AI to serve as a medium for social commentary.
"David Ha: We're going to see more of this. It's going to be gradual. Just like with any new technologies, when my kid grows up, maybe in a few years, they will take for granted a world where you can t..."
The Centaur Approach to Creativity
The conversation explores the centaur approach, where humans and AI collaborate to enhance creativity. Jim O'Shaughnessy shares insights on how artists can benefit from AI as a tool for iteration and feedback, fostering excitement about the future of art and technology.
"But what I noticed was people started to use these open-source models to communicate things that are culturally relevant at the time of the moments. In the US there's obviously lots of discussion and ..."
Understanding Large Language Models
David Ha provides a foundational overview of large language models, explaining their statistical nature and predictive capabilities. He discusses the evolution of these models and their significance in the context of machine learning, setting the stage for a deeper understanding of AI technologies.
"in Stability AI, is also right now negotiating with an AI tool company that allows the artists to actually work with their hand but then train it on their own dataset and keep a growing library of all..."
The Foundation of Machine Learning
David Ha explains the core principles of machine learning, emphasizing the importance of uncertainty in prediction models. He discusses how the evolution of data size has transformed machine learning from simple models to complex, high-dimensional neural networks capable of understanding intricate relationships within vast datasets.
"To the data point, you draw the best line that fits it and you observe some uncertainty around it, then this is a prediction model. Given X, you predict what likely Y is with some uncertainty. David H..."
Emergence of Deep Learning
Ha highlights the rise of deep learning over the past decade, illustrating how neural networks have evolved to learn from large datasets without extensive feature engineering. He reflects on the transition from traditional regression models to deep neural networks, drawing parallels to human brain function and the emergence of complex capabilities.
"exactly 10 years ago when researchers, some of my previous colleagues and Geoff Hinton has demonstrated that's these neural network models that can be trained to understand the statistical properties ..."
Self-Labeling Models
David Ha discusses the concept of self-labeling in machine learning, where models can learn from unlabelled data available on the internet. He explains how language models predict the next character or word, leading to emergent capabilities as the scale of the model increases, allowing for interactive and dynamic responses.
"So we're at the stage where neural network models are starting to be really good at prediction given, because it can model lots of data. David Ha: Then the interesting thing is, sure, you can train th..."
Emergent Phenomena in AI
Ha elaborates on the emergent phenomena observed in AI models, where simple objectives lead to complex outputs. He compares the evolution of AI capabilities to human civilization, suggesting that just as humans have emerged from basic survival instincts, AI can develop sophisticated functions from simple predictive tasks.
"as well. So before, maybe back in 2015, '16, when I was playing around with language models, you can feed it, auto Shakespeare, and it will blab out something that sounds like Shakespeare. David Ha: B..."
Causation vs. Correlation in AI
In this segment, David Ha addresses the complexities of causation versus correlation in AI systems. He contrasts traditional expert systems with emergent systems, emphasizing the challenges of ensuring that AI models do not misinterpret correlations as causations, using humorous examples from image classification.
"not planned but simply emerge from training on that objective. David Ha: It's similar as a researcher, I think both of us are interested in things like civilization and developments. We ourselves, we ..."
The Role of Data Structure
Ha discusses the significance of data structure in training AI models, particularly in the context of large language models. He explains how the inclusion of diverse datasets, including programming code, can enhance the model's ability to generate structured and methodical responses, impacting its overall performance.
"objective functions, live and pass our genes on. And out of those two simple objective functions we tried to maximize, came this incredible world of 8 billion sentient beings. So I love the connection..."
Challenges of Mathematical Accuracy
David Ha and Jim O'Shaughnessy explore the limitations of current AI models in performing mathematical tasks. They discuss the need for models to achieve high accuracy in numerical queries, emphasizing the challenges faced by CFOs and businesses in relying on AI for precise calculations.
"An emergent system would be like, okay, here are a million pictures of cats, figure out what's a cat. And it'll do that. David Ha: So it is very tricky because there's also the question of correlation..."
Integrating Tools with AI
Ha proposes a solution for improving AI's mathematical capabilities by integrating tools like calculators and databases into language models. He suggests that this approach could enhance the model's ability to provide accurate results and build user confidence in its outputs.
"One of my hypothesis is, it's a combination. Well, some works I did before, there's a paper called ‘Weight Agnostic Neural Networks’, where I tried to find... David Ha: My collaborators and I we did a..."
The Math Dilemma of Language Models
David Ha discusses the limitations of large language models in performing mathematical tasks. He compares their capabilities to human abilities, emphasizing that while humans struggle with precise calculations, they excel at generating reports and using tools like calculators. This segment explores the challenges of scaling language models to improve their mathematical accuracy and the potential of integrating them with external tools.
"are trained to blabber, text to predict the next characters and who knows how large of a scale they need to do until they're great mathematicians. As we mentioned earlier, if you want image model to g..."
Integrating Tools with Language Models
In this segment, David Ha explains the importance of training language models to use external tools, such as calculators and databases, rather than relying solely on their internal capabilities. He highlights companies like Adept AI and Perplexity AI that are pioneering this approach, allowing models to generate queries and interact with data effectively. This integration could significantly enhance the functionality of language models.
"let alone a language model at doing these things. David Ha: So getting back to the language model on the research and development side, there is quite a bit of work on not necessarily training these m..."
Emergence and Resource Constraints
David Ha elaborates on the concept of resource-constrained emergence in AI development. He argues that instead of merely scaling up models, we should focus on maximizing their capabilities within limited resources. This segment discusses how leveraging existing computational tools can lead to more efficient AI systems, drawing parallels to human evolution and the use of tools.
"out what queries they submitted to the database and also record what has been typed on a virtual calculator. David Ha: So, the user, the CFO can have confidence that, "Okay, this is the steps taken to..."
The Future of AI and Collaboration
In this discussion, David Ha and Jim O'Shaughnessy explore the future of AI, emphasizing the need for collaboration over competition. They reflect on the idea of cumulative cultural evolution and how AI can benefit from human-like adaptability. This segment highlights the importance of developing AI systems that can learn from and utilize tools effectively, fostering a collaborative environment.
"would just come back with my account. David Ha: Yeah. So, you can see how the magic of language models can be used in conjunction with so-called very strict ways, very methodical ways, mechanical ways..."
Incepting Humanity: A Call for Connection
David Ha shares his vision for a better world by encouraging communication and understanding among people. He suggests that individuals should engage with neighbors and spend quality time with children to foster connections. This segment emphasizes the importance of collaboration and understanding in both human interactions and AI development, aiming for a future where technology serves to enhance human relationships.
"Robert Anton Wilson calls that time-binding. David Ha: Yep. Jim O'Shaughnessy: And we see that this idea of the human colossus taking shape, where you move from strictly zero-sum games to positive-sum..."