"the following is a conversation with Dylan Patel and Nathan Lambert Dylan runs semi analysis A well respected research and Analysis company that specializes in semiconductors gpus CPUs and AI Hardware..."

"conversation is a deep dive into many critical aspects of the AI industry while it does get super technical we try to make sure that it's still accessible to folks outside of the AI field by defining ..."

"reveals its Chain of Thought reasoning which O3 mini does not it only shows a summary of the reasoning plus R1 is open weight and uh 03 mini is not by the way I got a chance to play with uh O3 mini an..."

"they will continue to shift the cost curve but the quote deep seek moment is indeed real I think it will still be remembered 5 years from now as a pivotal event in Tech History due in part to the geop..."

"we'll get into largely this is a open weight model and it's a instruction model like what you would use in chat GPT um they also release what is called the base model which is before these techniques ..."

"each of them there's so many places we can go here but maybe let's go to open weights first what does it mean for model to be open weights and what are the different flavors of Open Source in general ..."

"the software and what that means for AI is still being defined so uh for what I do I work at the Allen Institute for AI we're a nonprofit We want to make AI open for everybody and we try to lead on wh..."

"probably one of the more open models out of the frontier models so like in this full spectrum where probably the fullest open source like you said open code open data open weights this is not open cod..."

"for llama one of the most red p PDFs of the year last year is the Llama 3 paper but in some ways it's slightly less actionable it has less details on the training specifics like less plots um and so o..."

"chips I have never worked there myself and there are a few people in the world that do that very well and some of them are at Deep seek and these types of people are at Deep seek and leading American ..."

"Open Source so it's not the model that steals your data it's clovers hosting the model which could be China if you're using the Deep seek app or it could be perplexity uh you know you're trusting them..."

"that if you're an expert things that are close to The Fringe of knowledge they will still be fairly good at I think Cutting Edge AI topics that I do research on these models are capable for study Aid ..."

"open AI maybe it's useful here to go through like an example of a deep seek R1 reasoning yeah so the if if you're looking at the screen here what you'll see is a screenshot of the deep seek chat app a..."

"it's like it's reasoning through how humans feel emotions it's reasoning about meta emotions going to have pages and Pages this it's almost too much to actually read but it's nice to skim as it's comi..."

"pretty profound I mean you know this is AAL digression but a lot of people have found that these reasoning models can sometimes produce much more eloquent text that a at least interesting example I th..."

"more how are they able to achieve such low cost on the training and the inference maybe you could talk the training first yeah so there's there's two main techniques that they implemented that are pro..."

"and inference cost because now you're you know if you think about the parameter count as the sort of total embedding space for all of this knowledge that you're compressing down during training when y..."

"the Transformer is that useful let's go let's go into the Transformer the Transformer is a thing that is talked about a lot and we will not cover every detail uh essentially the Transformer is built o..."

"tasks a mixture of experts is one of the ones at training time even if you don't consider the inference benefits which are also big at training time your efficiency with your gpus is dramatically impr..."

"get into the details with this latent attention it's one of those things I look at it's like okay this they're doing really complex implementations because there's other parts of language model such a..."

"builds this Library called nickel right uh in which you know when you're training a model you have all these communications between every single layer of the model and you may have over 100 layers wha..."

"the mother of innovation and they had to do this whereas uh in the casa you know open AI has people that do this sort of stuff anthropic Etc uh but you know deep seek certainly did it publicly and the..."

"efficiency gains you get are not worth it but deep seeks implementation is so complex right especially with their mixture of experts right um people have done mixture of experts but they're generally ..."

"happens when a to you know this set of data that you get hey all of it looks like this one way and all of it should route to one part of my you know model right um so so when all of it rout routes to ..."

"which is this kind of lowlevel optimization or is this a shortterm thing where the biggest gains will be more on the algorithmic high level side of like posttraining is is this like a short-term leap ..."

"continue to drive success and therefore we were talking about relatively small implementation changes to the mixture of experts model and therefore it's like okay like we will need a few more years to..."

"readable code I think there is one aspect to note though right is that there is the general General ability for that to transfer across different types of runs right you may make really really high qu..."

"reality especially with more complicated stuff likee the biggest problem with it or FPA training which is another Innovation you know going to a lower Precision number format I.E less accurate is that..."

"Durk grenal has a theory A2 that's like Fast spikes and slow spikes where there are sometimes where you're looking at the loss and there other parameters you can see it start to creep up and then blow..."

"but how do you get if you're deep seek how do you get to a place where holy there's a successful combination of hyper parameters a lot of small failed runs and so so rapid uh iteration through failed ..."

"more no more around right uh no more screwing around everyone take all the resources we have let's pick what we think will work and just go for it right YOLO and this is where that sort of stress come..."

"the amount of computing time you have is is very low and you're you're you have to hit release schedules you have to not get blown past by everyone otherwise you know what happened with deep seek you ..."

"big Winners throughout human history are the ones who are willing to do yellow at some point okay uh what do we understand about the hardware it's been trained on deep seek deep seek is very interesti..."

"and papers saying like Hey we're the first company in China with an a100 cluster this large those 10,000 a100 gpus right this is this is in 2021 now this wasn't all for training you know large languag..."

"zoning in right like what do you call your training run right do you count all of the research and ablations that you ran right picking all the stuff because yes you can do a YOLO run but at some leve..."

"believe they actually have something closer to 50,000 gpus right now this is this is split across many tasks right again the fund um research in ablations for ballpark how much would open AI or anthro..."

"interconnect we can do all this fancy stuff to figure out how to use the GPU fully anyways right and and so that was back in October 2022 but uh later in 2023 end of 2023 implemented in 2024 the US go..."

"think this can be the goal of how some people describe export controls is this super powerful AI there's and you touched on the training run idea there's not many worlds where China cannot train AI mo..."

"much easier goal to achieve than trying to debate on what AGI is and if you have these extremely intelligent autonomous AIS and data centers like those are the things that could be running in these GP..."

"where like AI is T is like measured in the power of in like how much power is delivered to compute right or how much uh is being you know that's sort of a way of thinking about what's the economic out..."

"take off in the US Open AI needs a ton of gpus on inference to capture this they have this um open AI chat gbt Pro subscription which is $200 a month which Sam said they're losing money on which means..."

"the cost of actually operating that capability yeah this is going to be my point so so extreme that no one can actually deploy it at scale and Mass to actually completely revolutionize the economy on ..."

"dollars of GPU time and there just won't be enough power gpus infrastructure to operate this and therefore shift everything in the world on the snap the finger but at that moment who gets to constu co..."

"everything I've seen uh people's intuition seems to fail on robotics so you have this kind of General optimism I've seen this on self-driving cars people think it's much easier problem than it is simi..."

"there could be cyber War cyber War type of technologies that uh from social engineering to actually just swarms of robots that find attack vectors in our code bases and shut down P grids that kind of ..."

"compute because Talent is not really something that's constraining right China arguably has more Talent right more stem graduates more programmers the US can draw upon the world's people which it does..."

"exactly to the the manufactur stuff so why why so longterm they're going to be manufacturing chips there chips are a little bit more specialized I'm specifically referring to the data centers right ch..."

"medium short term right um and so that's that's the big unlocker there um and even even today right if xingping decided to get you know quote unquote scale pilled right uh I.E decide that scaling laws..."

"they only just released a subsidy of a trillion R&B uh you know roughly $160 billion um which is close to the spending of like Microsoft and meta and Google combined right for this year so it's like t..."

"so you look at a Leading Edge Fab that is going to be profitable today that's building you know three nanometer chips or two nanometer chips in the future that's going to cost north of 3040 billion ri..."

"right including Intel their latest PC chip uses tsmc chips right it also uses some Intel chips but it uses tsmc process can you explain why the foundry model is so successful for these companies why w..."

"types of chips you're not going to can have the demand to pay back the cost of the Fab whereas Nvidia can have many different customers and aggregate all this demand into one place and then they're th..."

"they've just been the best right they are so good at it right they're customer focused they make it easy for you to fabricate your chips they take all of that complexity and like kind of try and Abstr..."

"of the world right um so so there is there is a large dichotomy of like what is the top 1% of the society doing and where are they headed because of economic reasons right Intel never paid that crazy ..."

"when you talk about hey you have all these people that are super specialized they will work you know 80 hours a week in a factory right in a Fab and if anything goes wrong they'll go show up in the mi..."

"better than everyone the tool guys were like oh I don't think that this this is mature enough and they're like ah you just don't know we know right this sort of stuff would happen um and so can the US..."

"really is fundamentally about R&D and it is all about tsmc huh and so tsmc you know you cannot purchase a vehicle without tsmc chips right you cannot purchase a fridge without tsmc chips you cannot yo..."

"nanm power IC or analog IC or you know random chip in my keyboard right that kind of stuff so so there is an angle of like the US's actions have been so from these export you know from the angle of th..."

"dollars right and so like the amount of money that the US is spending on the semiconductor industry is is nothing right um whereas all these other countries have uh structural advantages in terms of l..."

"the equation for tsmc building more Fabs in the US that's what he's sort of positing right so can you lay out the so we laid out the importance by the way it's incredible how much you know about so mu..."

"of AI I mean ultimately the export controls are pointing towards a separate future economy I think the US has made it clear to Chinese leaders that we intend to control this technology at whatever cos..."

"I have zero idea and I would love if we Kum we could all hold hands and sing Kumbaya but like I have zero idea how that could possibly happen is the Divergence good or bad for avoiding war is it possi..."

"right the last hand you know decades now we we've sort of seen things start to slide right with Russia Ukraine with what's going on in the Middle East and you know Taiwan risk all these different thin..."

"position and therefore I I hope that works and and and as an American like you know kind of like okay I guess that's going to lead to peace peace for us uh now obviously other people around the world ..."

"h20s promising yeah so this goes uh and I think we'd have to like we need to dive really deep into the reasoning aspect and what's going on there but the H20 you know the US has gone through multiple ..."

"lot of compute they involve a lot of moving memory around uh whether it be to memory or two other chips right and so these three vectors um the US initially had a multi you know had two of these vecto..."

"we at our research we cut nvidia's production for H20 for this year down drastically they were going to make another 2 million of those this year but they just canel all the orders a couple weeks ago ..."

"right we talk about models in terms of like how many flops they are right uh so so like you know we talk about oh gp4 is 2 e25 right two to the uh two to the 25 fth uh you know 25 Z right flop right f..."

"of this humongous revolution in the last handful of years is the Transformer right and the attention mechanism attention mechanism is that the model understands the relationships between all the words..."

"context length were like let me put a ton of documents in and then get an answer out right and it's a it's a single you know pre-fill compute a lot in parallel and then output a little bit now with re..."

"right whereas with reasoning I'm now generating tens of thousands of tokens in in sequence right and so this this memory this KV cach has to stay resident you have to keep loading it you have to keep ..."

"cheap talk about why it's so cheap on the inference it works well and it's cheap why is R1 so damn cheap so I think there's a couple factors here right one is that they do have model architecture Inno..."

"relative to Prior forms all right that's the memory pressure I should say in case people don't know R1 is 27 times cheaper than 01 we think that open AI had a large margin built in okay so that's ther..."

"don't actually think so um and part of that is this chart right look at all the other providers right together AI fireworks AI are very highend companies right xmeta together AI is treow and the inven..."

"some of the interviews there's discussion on how like doing this is a recruiting tool you see this at the American companies too it's like having gpus recruiting tool being at The Cutting Edge of AI r..."

"why anthropic is not open sourcing things that's their claims but there's reviews internally anthropic um ra mentions things to International governments there's been news of how anthropic has done pr..."

"the US companies right this is something that Dario talks about is like that's the situation that Dario wants to avoid is Dario talks to about the difference between race to the bottom and race to the..."

"this publicly in an instruct model that's open weights this can then proliferate right but as these systems get more and more capable what you can embed deep down in the model is not as clear right um..."

"boil down into like very very important topics like hey you know sub you know subverting people right uh you know chat Bots right character AI has shown that they can like you know talk to kids and or..."

"people very we don't know the extent that which people can be impacted by that so there there could be this is one this is an actual concern with a Chinese company that is providing open weights model..."

"poisoned the pre-training data I don't think like as of now I don't think anybody in a production system is trying to do anything like this I think it's mostly anthropic is doing very direct work and ..."

"more more on these kinds of systems I mean we've already seen this with recommendation systems yeah recommendation systems hack the the dopamine induced reward circuit but the brain is a lot more comp..."

"be not other people but algorithms or other people presented to me via algorithms there I mean there are already tons of AI bots on the internet and every so right now it's not frequent but every so o..."

"companies that deploy them so one case when we've seen that and maybe censorship is one word alignment maybe via rhf or some other way is another word so that we we saw that with black Nazi image gene..."

"by the way removing facts has such an ominous dark feel to it almost think it's practically impossible because you effectively have to remove them from the internet you're you're taking on a did did d..."

"always have some TDS Trum derangement syndrome because it's trained so much it'll have the ability to express it but what if what if you there's a there's a wide representation in the data this is wha..."

"we can go through multiple examples and what happened llama 2 was a launch that the phrase like too much rhf or like too much safety was a lot it's just that was the whole narrative after llama 2's ch..."

"rhf that it makes the models dumb and it stigmatized the word it did in AI culture and as the techniques have evolved that's no longer the case where all of these Labs have very fine grain control ove..."

"comparing I would say highest cost and highest total usage so a lot of money has gone to these Wiz comparisons where you have two model outputs and a human is comparing between the two of them in earl..."

"this might be a good place to uh to mention the uh the eloquent and the insightful tweet of the Great and The Powerful Andre kathi uh I think he had a bunch of thoughts but one of them last thought no..."

"of the process was learning from humans where they had they started the first this is the first expert level go player or chess player in Deep Mind series of models where they had some human data and ..."

"efficient that is because of the self-play right how does a baby learn what its body is is it sticks its foot in its mouth and it says oh this is my body right it sticks its hand in its mouth and it c..."

"this is where I think the like aha moment of computer use or robotics will come in because now you have a Sandbox or a playground that is infinitely verifiable right did you you know messing around on..."

"approach math with language models just by increasing the number of samples so you can just try again and again and again and you look at the amount of times that the language models get it right and ..."

"deeps paper detailed in this R1 paper which for me is one of the big open questions on how do you do this is that they did reasoning heavy but very standard post trining techniques after the large sca..."

"yeah it's cool and it's revealing uh the reasoning it's it's magical it's magical like this is really powerful hello everyone this is Lex with a quick intermission recorded after the podcast since we ..."

"element for each of these models is how the reasoning is presented deep seek R1 shows the full Chain of Thought tokens which I personally just love for these open-ended philosophical questions it's re..."

"domestication by choice is a really interesting angle again it's one of those things when somebody presents a different angle on a seemingly obvious thing it just makes me smile and the same with deep..."

"an intrinsic cognitive process that acts like an internal error correction system it allows us to adapt our identities and values over time in response to new experiences challenges and social context..."

"and then and and and what o1 offers and then open AI has 01 Pro and what they did with 03 which is like also very unique is that they stacked search on top of Chain of Thought right um and so Chain of..."

"more in multiple choice is what they're doing or if it's something more complex where they Chang the training and they know that the inference mode is going to be different so we're talking about 01 P..."

"they're not below the trend line first of all and at least for gpt3 right uh they are the first to hit it right which is which is a big deal um but they're not below the trend line as far as gpt3 now ..."

"that is Nvidia stock plummeted uh can you explain what happened I mean and also just explain this moment and whether you know if Nvidia is going to keep winning we're both Nvidia Bulls here I would sa..."

"there's also an element of Nvidia has just been a straight line up right and and there's been so many different narratives that have been trying to push down Nvidia not I don't say push down Nvidia st..."

"that does everything reliably right now because it's not like an Nvidia competitor arose it's it's another company that's using Nvidia who historically has been a large Nvidia customer customer yeah a..."

"smuggling most of the large scale smuggling is like companies in Singapore and Malaysia like routing them around or renting gpus completely legally I want to jump in how much do the scale I think ther..."

"massive network of like companies to get the materials they need after they were banned in like 2018 so it's not like otherworldly uh but I agree right n Nathan's point is like hey you can't smuggle A..."

"smuggle but yeah it's not you know as the numbers grow right uh you know 100 something billion dollars of revenue for NVIDIA last year 200 something billion this year right and if next year or you kno..."

"would be fascinating to watch the smuggling cuz I mean there's drug smuggling right that's a that's a market there's weapons smuggling and gpus will surpass that at some points are highest value per k..."

"have some completions what the model is trying to learn to imitate and what you do there is instead of a human data or instead of the model you're currently training you take completions from a differ..."

"right so there's a bit of a hypocrisy because sort of open Ai and potentially most of the companies trained on the internet's text without permission there's also a clear loophole which is that uh I g..."

"chbt oh I guess I guess one of the ways to do that is like a system prompt or something like that like if you're serving it to say that you're that's what that's what we do like if we host the demo yo..."

"the last couple days we've seen a lot of people distill deep seeks model into llama models because because the Deep seek models are kind of complicated to run inference on because their mixture of exp..."

"clusters and what's a GPU and what's a computer and what kid not that far back but yeah so like what do we mean by the Clusters I thought I was about to do the Apple ad right what's a computer so so t..."

"millions of gpus but the scale of the uh largest cluster is also really important right um when we look back at history right like you know or through through the age of AI right like it was a really ..."

"toaster toaster is like a similar power consumption to an a100 Right h100 comes around they increase the power from like 400 to 700 watts and that's just per GPU and then there's all the associated st..."

"know think about 100,000 gpus um with roughly 1,400 Watts a piece that's that's that's 140 megawatts 150 megawatts right for 128 right so you're talking about you've jumped from 15 to megawatts to 10x..."

"right like all all the hypers scalers have done this now the next scale is is is something that's even bigger right and so you know Elon just to stick on the topic he's he's building his own natural g..."

"right they're building two natural gas plants massive ones uh and they're and then they're building this massive data center um Amazon has like plans for this scale uh Google has plans for this scale ..."

"in some parts of the US like in Virginia it cost more to transmit power than it cost to generate it which is like you know there's there's all sorts of like second order effects that are insane here c..."

"couple hopes right like one is you know and Elon what he's doing in Memphis is like you know to the extreme they're not just using dual combine cycle gas which is like super efficient he's also just u..."

"lose we you know that's way worse right I should say that uh I got a chance to visit um the Memphis data center oh wow and it's uh kind of incredible I mean I visited with with Elon just the team them..."

"the unsung heroes are the cooling and electrical systems which are just glossed over um but I think like like one story that maybe is like exemplifies how insane this stuff is is uh when you're traini..."

"power doesn't Spike but that just tells you how much power you're working with I mean it's insane it's insane people should just go Google like scale like what does X watts do and go through all the s..."

"section called cluster measuring contest so uh there's another word there but I won't say it you know uh what who's who's who's got the biggest now and who's going to have the big today individual lar..."

"right you know that's he's going to surprise us so what's the idea with these clusters if you have a million gpus what percentage in uh let's say two three years is used for uh training and what perce..."

"pre-training is when you increase the context length for these models and we've talked earlier in the conversation about how the context length when you have a long input is much easier to manage than..."

"to have the biggest cluster fully connected right because it's all in one building yeah right and he's completely right on that right Google has the biggest cluster but you have to spread over three s..."

"I mean like you're always going to make more money on Services than than than I mean like yeah like you like to be clear like today people are spending a lot more on Hardware than they are the service..."

"Nvidia it should be said as a truly special company like I mean they the whole the culture of everything they're really optimized for that kind of thing speaking of which is there somebody that can ev..."

"think I think the you know people focus on the payback question but it's really easy to like just be like well like you know GDP is humans and Industrial Capital right and if you can make intelligence..."

"okay so it's not uh let's not call it AGI whatever it's like a single day it's it's a gradual thingi super powerful AI but it's it's a gradually increasing set of features that are useful and uh make ..."

"already sell so whether that's the recommendation system or for Elon who's been talking about Optimus the robot potentially the intelligence of the robot and then you have personalized robots in the h..."

"brand is in Chachi PT and there is actually not that for most users there's not that much of a reason that they need open AI to be spending billions and billions of dollars on the next best model when..."

"the only use case it's like these reasoning code agents computer use all this stuff is where opena has to actually go to make money in the future otherwise they're kaputs but X Google and meta have th..."

"of discussions as it's the next compute layer you you have to believe that and and you there's a lot of discussions that tokens and tokenomics and llm apis are the next compute layer or or the next Pa..."

"commodity right deeps V3 shows this but also the gpt3 3 chart earlier C chart showed this right llama 3B is 1 1200X cheaper than gpt3 any gpt3 like anyone whose business model was gpt3 level capabilit..."

"is totally untapped and it's not clear technically how it is done yeah that is I mean the sort of the AdSense Innovation that Google did the one day you'll have in GPT output an ad and that's going to..."

"AGI yeah agents and AGI and if I build AGI I can make tons of money right or I can spend pay for everything right and this is this is It's just predicated like back on the like export control thing ri..."

"sandboxes already exist in research there are people who have built clones of all the most popular websites of Google Amazon blah blah blah to make it so that there's and I mean open AI probably has t..."

"think about the programming context so software engineering that you know that's where I personally and I know a lot of people um interact with AI the most there's a lot of fear and angst too from cur..."

"because it is a verifiable domain um you can always like unit test or compile um and and and there's many different regions of like it can inspect the whole code base at once which no no engineer real..."

"all of these things can go happen fast I think software and then and then the other domain is like industrial chemical mechanical engineers suck at coding right uh just generally and like their tools ..."

"the nature of what it means to be a programmer and what kind of jobs programmers do changes because I think there needs to be a human in the loop of everything you've talked about there's a really imp..."

"else oh yeah there's so many lwh hanging fruit everywhere in terms of where software can like help automate a thing or digitize the thing in in the legal system I mean that's why doge is exciting you ..."

"available but it's like posttraining is much more accessible at this time it's still pretty cheap and you can do it and the thing is like how high can we push this number where people have access to a..."

"model and this model we released today we saw the same thing is we're at ai2 we don't have a ton of compute we can't train 405b models all the time so we just did a few runs and they tend to work and ..."

"release a model later we have more time to learn new techniques like this RL Technique we had started this in the fall it's now really popular reasoning models the next thing to do for open open sourc..."

"terabytes of files it's like I I I don't know what I'm going to find in there but that's what that's what we need to do as an ecosystem if people want open source AI to be financially useful we didn't..."

"predicated and this is why that whole show happened now how they came up with a $500 billion number is beyond me how they came up with a hundred billion dollar number makes sense to some extent right ..."

"furthermore it's not $100 billion it's $50 billion of spend right and then like $50 billion of operational cost power Etc um rental pricing Etc um because they're renting it from opening eyes is renti..."

"have to do with anything what does Trump have to do with everything he's just a hype man Trump is he's reducing the regulation so they can build it faster right um and he's allowing them to do it righ..."

"could be very specific technical things like breakthroughs on post post training or it could be just size big yeah I mean it's impressive clusters I really I really enjoy tracking supply chain and lik..."