All Segments
Why Scale Will Not Solve AGI | Vishal Misra - The a16z Show
8 segments available
Vishal Misra returns to explain his latest research on how LLMs actually work under the hood. He walks through experiments showing that transformers update their predictions in a precise, mathematically predictable way as they process new information, explains why this still doesn't mean they're conscious, and describes what's actually required for AGI: the ability to keep learning after training and the move from pattern matching to understanding cause and effect. Timestamps 00:00 — Introduction 02:58 — LLM as Giant Matrix 08:24 — What Is In-Context Learning 13:00 — Bayesian Updating as Evidence 19:13 — Bayesian Wind Tunnel Tests 27:22 — Brains Simulate Causality 36:34 — Manifolds and New Representations 42:17 — Simulation as Short Program Read the full transcript here: https://www.a16z.news/s/podcast Resources: Follow Vishal Misra on X: https://x.com/vishalmisra Follow Martin Casado on X: https://x.com/martin_casado Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures.
Segments Timeline
Introduction
"Anthropic makes great products. Clot code is fantastic. Co-work is fantastic. But they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monolo..."
LLM as Giant Matrix
">> you were trying to you were trying to describe you're trying to come up with a mathematical model of how LLM works. >> Yeah. >> And you had which was very helpful to me which was um and at the time..."
What Is In-Context Learning
">> right >> you you know you use this approach to describe how in context learning works and so maybe first describe what in context learning is and then kind of the conclusion that you came from that..."
Bayesian Updating as Evidence
">> Yeah. So, so when you think about what in context learning is is that as you see evidence. So, so you know in the first paper what I also did was I I took this cricket DSL example. >> Yeah. >> And ..."
Bayesian Wind Tunnel Tests
"this idea you know my colleagues at Namanagaral and Sedhad Dalal we the series of papers were were written with them. We came up with this idea of a Beijian wind tunnel. Okay so what's a wind tunnel? ..."
Brains Simulate Causality
">> Beijian, >> but we do something else. You know when I when I when I throw this pen at you, what will you do? >> Dodge it or >> do it? Yeah. >> Why will you dodge it? >> To avoid being hit. >> Avoid..."
Manifolds and New Representations
"Yeah. You know, another way that I've always thought about these, I thought you articulated it well in the last time we talked about it, which is the universe is this very, very complex space and then..."
Simulation as Short Program
">> Can you and can you can you tie the two things like how does that pair with doing simulation or is that simulation totally orthogonal? >> No, simulation is is related, right? So you think it like b..."