All Segments
Recursion Is The Next Scaling Law In AI
12 segments available
A 7-million parameter model outperforming models a thousand times its size on tasks like ARC Prize. That's what recursive reasoning unlocks. In this episode of Decoded, YC's Ankit Gupta and Francois Chaubard break down two recent papers on recursive AI models, HRMs and TRMs, that are achieving state-of-the-art results with a fraction of the parameters of today's largest models. They explain why standard LLMs hit a fundamental ceiling on certain reasoning tasks, how recursion at inference time gives small models the compute depth to break through it, and what happens when you combine these ideas with the power of large-scale foundation models. Apply to Y Combinator: https://www.ycombinator.com/apply Work at a startup: https://www.ycombinator.com/jobs 00:00 - Intro 00:35 - Model Foundations 01:15 - RNN Limits and LLM Contrast 02:36 - Reasoning Limits and Sorting Analogy 04:22 - HRM Paper Introduction 05:25 - HRM Architecture and Intuition 07:36 - HRM Results and Outer Loop 09:46 - TRM Paper Overview 11:20 - TRM Training and Fixed Point 13:30 - Detailed HRM Summary 20:46 - Comparing HRM and TRM 34:45 - Future Outlook and Outro
Segments Timeline
Intro
"Welcome back to another episode of Decoded. Today, I'm back with YC visiting partner Francois Shaard to talk about one of the most interesting recent trends in AI research, recursion. Specifically, we..."
Model Foundations
"Franis, thanks for joining us. Um, can you tell us a little bit about these two models and what was so interesting about them? >> Sure. I guess, um, to set up a little bit of a foundation, uh, you alr..."
RNN Limits and LLM Contrast
">> So this is about 10 years ago people were working on these models. This was in the era of LSDMs and LSTMs with attention. >> Yeah. And uh depending which professors you talk to uh before attention ..."
Reasoning Limits and Sorting Analogy
"they're doing basically this one on oneot feed forward process for every input right the the LLM the transformer block can take all the inputs in parallel. It's not actually iteratively going over the..."
HRM Paper Introduction
"like we just watched val loss go down, perplexity goes down, like the model just is more performant, looks better, starts to make some Shakespeare that actually sounds somewhat plausible, right? >> An..."
HRM Architecture and Intuition
"so are mazes. Those are incompressible problems. Rolling sum incompressible problem. So when you mention the sorting algorithm, when I think back to my algorithms class from college, the one way you c..."
HRM Results and Outer Loop
"used to. So, let's talk about HRM first. Um, walk me through a little bit about how this model works and some of the intuition behind it. >> Sure. So um the the the this is directly in the lineage of ..."
TRM Paper Overview
"produce an effective paper result and what even were the results that this paper showed? >> Yeah. And so I mean this got state-of-the-art um on arc prize uh one and two. uh this was a only a 27 millio..."
TRM Training and Fixed Point
"get around that. So like what was that trick that they figured out? And this is really the the crux of the paper that like differentiates it in my opinion in the literature is they instead of doing wh..."
Detailed HRM Summary
">> like different memory states basically. It's like across this um hidden or carry memory access basically >> and and that math holds and it works. it follows DEQ directly in the event that the ZL an..."
Comparing HRM and TRM
"here. What matters most from the HRM paper that we should take away before we transition and contrast it with the TRM paper. >> Yeah, I think that the the number one piece uh to take away is this oute..."
Future Outlook and Outro
"people think about where these models fit into the current span of research happening, especially given that it seems like a bit of a departure from a lot of the methods that people are used to hearin..."