"Today I'm chatting with Sergey Levine, who  is a co-founder of Physical Intelligence,   which is a robotics foundation model company,  and also a professor at UC Berkeley and just   generally one of t..."

"Physical Intelligence aims to  build robotic foundation models.  That basically means general-purpose  models that could in principle   control any robot to perform any task. We care about this becaus..."

"It's just putting in place the basic  building blocks, on top of which we can   then tackle all these really tough problems. What's a year-by-year vision? One year in,   I got a chance to watch some o..."

"As you mentioned, folding a box, folding different  articles of laundry, cleaning up a table,   making a coffee, that sort of thing. That's  good, that works. The results we've been   able to show are..."

"I like to do my laundry on Saturday, so make  sure that it's ready. This and this and this.   By the way, check in with me every Monday to  see what I want you to pick up when you do the   shopping." ..."

"There's a lot more that goes into this. But the principles there are:   you need to leverage prior knowledge and  you need to have the right representations.  This grand vision, what year? If you had ..."

"care about, that you would want to see… I don't  know but single-digit years is very realistic.  I'm really hoping it'll be more like one or  two before something is actually out there,   but it's har..."

"many organizations are working on exactly this. In fact, arguably there is already a flywheel.  It’s not an automated flywheel  but a human-in-the-loop flywheel.  Everybody who's deploying an LLM is o..."

"I don't think there's a profound  reason why robotics is that different.  There are a few small differences that  make things a little bit more manageable.  Especially if you have a robot that's doing..."

"It's actually not that different from what we've  seen with LLMs in some ways. It's a matter of   scope. Think about coding assistants.  Initially the best tools for coding,   they could do a little b..."

"rather than a double-digit thing. The reason it's hard to really pin   down is because, as with all research, it does  depend on figuring out a few question marks.  My answer in terms of the nature of..."

"able to do most blue-collar work in the economy. There's a nuance here. It becomes more obvious if   we consider the analogy to coding assistants. It's not like the nature of coding assistants   today..."

"what LLMs are in now, or is it more like we have  robots deployed everywhere and they're actually   doing a whole bunch of real work, et cetera? It's a very subtle question. What it probably   will co..."

"But there's so many things  which increase productivity.  Like wearing gloves increases  productivity or I don't know.  You want to understand something which  increases productivity a hundredfold   v..."

"it's also learning from words. Eventually it’ll be learning   from observing what people do from the kind of  natural feedback that you receive when you're   doing a job together with somebody else. T..."

"will depend on our ability to handle all of the  complex edge cases and handle them correctly.  Now construction is something where in  some places it's extremely delicate,   where you really have to ..."

"is a very reasonable timeline for that. Just to better explain the parameters   of this, scope means a few things. Scope means how much responsibility   you're willing to delegate to the machine. But ..."

"The reason I care about this so much is…obviously  automation is good, productivity is good.  But I also care about this because the more  these things get out there, the closer we are   to getting th..."

"In terms of robotics progress,  why won't it be like self-driving cars, where it's been more than 10 years since Google   launched its… Wasn't it in 2009 that they  launched the self-driving car initi..."

"To give you an example, if you're learning  how to drive, you would probably be pretty   crazy to learn how to drive on your  own without somebody helping you.  You would not trust your teenage child ..."

"it's not just a laboratory science experiment. It also requires industrial scale building effort.  It's more like the Apollo program  than it is a science experiment.  The excellent research that was ..."

"Just to give an order of magnitude, how  does the amount of data you have collected   compare to internet-scale pre-training data? I know it's hard to do a token-by-token count,   because how does vid..."

"How does the π0 model work? The current model that we   have basically is a vision-language model  that has been adapted for motor control.  To give you a little bit of a fanciful brain  analogy, a VL..."

"One theme here that is important to keep in mind  is that the reason that those building blocks   are so valuable is because the AI community has  gotten a lot better at leveraging prior knowledge.  A..."

"I was talking to this researcher, Sander at  GDM, and he works on video and audio models.  He made the point that the reason, in his  view, we aren't seeing that much transfer   learning between diffe..."

"You had a really interesting blog post about why  video models aren't as robust as language models.  Sorry, this is not a super well-formed question. I just wanted to get a reaction.  Yeah, what’s up ..."

"Whereas with language, clearly it has. This point about representations   is really key to it. One way we can think about it is this.  Imagine pointing a camera outside this building,  there's the sky..."

"The representations are already there.  They're not just good representations,   they focus on what really matters. That's the  bad news. Here's the good news. The good news   is that we don't have to..."

"The fact that video models aren't as  robust, is that bearish for robotics?  So much of the data you will have to use… I  guess you're saying a lot of it will be labeled.  Ideally, you just want to be..."

"robots really does help with generalization. I have the suspicion that in the long run,   it'll make it easier to use those sources of  data that have been tricky to use up until now.  Famously, LLMs ..."

"out-of-distribution capabilities. I wonder if the trek over the next   5-10 years will be like this: Each subtask,  you have to give it thousands of episodes.  Then it's very hard to actually automate..."

"Or is there some reason to think that it  will progress more generally than that?  There's a subtlety here. Emergent  capabilities don't just come from the   fact that internet data has a lot of stuff..."

"That's actually where the  emergent capabilities come from.  Because of this, in principle, if we  have a sufficient diversity of behaviors,   the model should figure out that those  behaviors can be ..."

"For the kind of robot which is just cleaning  up your house, I think it has to be aware of   things that happened minutes ago or hours  ago and how that influences its plan   about the next task it's ..."

"try to unpack this a little bit. There's a lot going on in there.  One thing is a really  interesting technical problem.  It's something where we'll  see perhaps a lot of really   interesting innovati..."

"Do you mean in terms of how we represent? How we represent both context,   both what happened in the past, and also plans or  reasoning, as you call it in the LLM world, which   is what we would like ..."

"That's a really good question. I  definitely don't know the answer to this.  I am not by any means well-versed in neuroscience. If I had to guess and also provide an answer that   leans more on things..."

"If in five years we have a system  which is as robust as a human in   terms of interacting with the world, then  what has happened that makes it physically   possible to be able to run those models? T..."

"In this robotics world, should we  just be anticipating something where   you need connectivity everywhere? You need robots that are super fast.  You're streaming video information back and  forth, or..."

"Again, this is not a new idea. This is exactly what we've seen with LLMs.  LLMs start off being trained  purely with next token prediction.  That provided an excellent starting  point, first for all s..."

"et cetera, which might be different from  the considerations for knowledge work.  Maybe it's still the same model, but  then you can serve it in different ways.  The advantages of co-training are high..."

"Once we have, in 2035 or 2030, basically this  sci-fi world, are you optimistic about the   ability of true AGIs to build simulations in  which they are rehearsing skills that no human   or AI has eve..."

"Do you have a sense of what  the equivalent is in humans?  Whatever we're doing when  we're daydreaming or sleeping.  I don't know if you have some sense of  what this auxiliary thing we're doing is, ..."

"You can just crunch numbers on having  100-200 gigawatts deployed by 2030.  The marginal capex per year is  in the trillions of dollars.  It's $2-4 trillion dollars a year. That corresponds to actual ..."

"It can be a big productivity boost for real  people and it can allow you to solve problems   that are very difficult to solve. For example, I'm not an expert on   data centers by any means, but you co..."

"Interesting. Do you think the  learning rate will continue?  Do you think it will cost hundreds of dollars  by the end of the decade to buy mobile arms?  That is a great question for my co-founder, Ad..."

"demand when there's a lot of demand. How many iPhones were in the world in   2001? There's definitely a challenge there.  It's something that is worth thinking about.  A particularly important questio..."

"One broader theme here is that if you want to  have an economy where you get ahead by having   a highly educated workforce—by having people  that have high productivity, meaning that   for each person..."

"All of that stuff is pretty complicated. It requires making a number   of really good decisions. Good decisions about investing in   a balanced robotics ecosystem, supporting both  software innovation..."

"But I'm very optimistic about it because  it seems to me that the light at the end   of the tunnel is in the right direction. I guess there's a different question.  If the value is bottlenecked by har..."

"it seems like a different question than, "Well,  what is the impact on human wages or something?"  For the specifics of how we make that happen,  that's a very long conversation that I'm probably   no..."

"It's not necessarily bad that they help others. But to the extent that a lot of the things   which would go into this feedback loop—the  sub-component, manufacturing and supply chain,   already exist ..."

"holistic view of these things. I wish we had more holistic   conversations about that sometimes. From the perspective of society as a whole,   how should they be thinking about the  advances in roboti..."

"evolves quite the way that people expect. Sometimes the journey is just as important   as the destination. It's very difficult   to plan ahead for an end state. Directionally, what you said makes a lo..."