
47 segments available
The AI hardware race is heating up, and NVIDIA is still far ahead. What will it take to close the gap? In this episode, Dylan Patel (Founder & CEO, SemiAnalysis) joins Erin Price-Wright (General Partner, a16z), Guido Appenzeller (Partner, a16z), and host Erik Torenberg to break down the state of AI chips, data centers, and infrastructure strategy. We discuss: - Why simply copying NVIDIA won’t work, and what it takes to beat them - How custom silicon from Google, Amazon, and Meta could reshape the market - The economics of AI model launches and the shift toward cost efficiency - Infrastructure bottlenecks: power, cooling, and the global supply chain - The rise of AI silicon startups and the challenges they face - Export controls, China’s AI ambitions, and geopolitics in the chip race - Big tech’s next moves: advice for leaders like Jensen Huang, Sundar Pichai, Mark Zuckerberg, and Elon Musk Timecodes: 0:00 Introduction & AI Hardware Landscape 1:11 Reactions to GPT-5: Is It Disappointing? 4:19 The Business of AI Models: Cost, Monetization, and the Router 7:34 The Economics of AI: Cost vs. Performance 10:10 Usage-Based Pricing & Product Stickiness 12:30 Advice for Sam Altman: Monetizing OpenAI 14:18 NVIDIA’s Growth & The Future of AI Compute 21:27 Custom Silicon: Threats to NVIDIA 26:09 The Silicon Startup Boom 45:28 Data Center Power & Cooling: The Next Bottleneck 50:46 Intel’s Role in the AI Era 57:56 Advice for Tech Giants: NVIDIA, Google, Meta, Apple, Microsoft 1:08:17 AI Policy & Export Controls Resources: Find Dylan on X: https://x.com/dylan522p Find Erin on X: https://x.com/espricewright Find Guido on X: https://x.com/appenz Learn more about SemiAnalysis: https://semianalysis.com/dylan-patel/ Stay Updated: Let us know what you think: https://ratethispodcast.com/a16z Find a16z on Twitter: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Subscribe on your favorite podcast app: https://a16z.simplecast.com/ Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details, please see a16z.com/disclosures.
Dylan Patel discusses the competitive edge NVIDIA holds in the AI hardware landscape, emphasizing their superior networking, cost efficiency, and faster market ramp-up. He argues that simply replicating NVIDIA's approach won't suffice; companies must innovate significantly to compete.
"Nvidia is going to have better networking than you. They're going to have better uh HPM. They're going to have better process node. They're going to come to market faster. They're going to be able to ..."
The hosts introduce Dylan Patel, highlighting his expertise in AI hardware and semiconductor analysis. They express excitement about discussing the current state of AI chips and the implications for the data center market.
"super excited to have you here today. Awesome. Thank you. Uh happy to talk about my favorite topics. Amazing. Well, maybe let's start with GP5. You know, we just had the um some of the researchers fro..."
Dylan shares his initial reactions to GPT-5, noting that it may be disappointing for some users. He compares it to previous models, discussing the differences in performance and user experience, particularly regarding compute efficiency.
"of user you are, right? Um, and so like right if you're just using GPD5 and before you were, you know, $20 or $200 a month subscriber, um, you no longer have access to 4.5, which in my opinion is stil..."
Dylan explains the new router functionality in GPT-5, which optimizes how queries are processed. He discusses its implications for compute allocation and user experience, suggesting it could be a significant step in monetizing free users.
"you think about you know what what is this curve of intelligence right it's like the more compute you spend the better the model gets um and that's whether it's a bigger model uh which GPD5 isn't righ..."
The conversation shifts to the economics of AI models, focusing on the importance of cost efficiency. Dylan highlights how OpenAI's approach to pricing and model usage reflects a growing trend towards balancing cost and performance in AI applications.
"less compute going into uh a power user's average query than before. But isn't it even more interesting? OpenAI can now control how much comput wants to allocate to you, right? if if if we're in a hig..."
Dylan discusses strategies for monetizing free users of AI models, emphasizing the need for innovative approaches that don't compromise user experience. He suggests that routing queries based on value could be a key to generating revenue from free users.
"uh if they need to right and it's like I think I think the router points to the future of open eye from a business right like you can look at sort of the model companies right anthropic is fully focus..."
The discussion continues on how the AI landscape is shifting towards cost-based metrics. Dylan notes that the focus is moving from purely performance metrics to a balance of cost and performance, which is becoming crucial for competitive advantage.
"enough it'll be able to you know contact all the lawyers in the area and um you know figure out what their results are and maybe search their like court filings and whatever right and and and book the..."
Dylan analyzes user behavior in relation to pricing models for AI tools. He highlights the challenges of usage-based pricing and the potential for subscription models to provide more predictable revenue streams for AI companies.
"longer where cost alone is is is that what we're seeing here or I mean I think definitely right like OpenAI said they doubled their rate limits for uh you know c for big amounts of uh users they've th..."
The conversation shifts to the importance of designing effective user feedback loops in AI systems. Dylan emphasizes that the interaction between model performance and user input is critical for improving AI tools and enhancing user satisfaction.
"account. Is that the future then? Um I mean but it's it's clear like people are taking advantage of the negative gross margin like sort of uh you know subscriptions that are offered. Um you know I thi..."
Dylan is asked for advice on how OpenAI can increase its value. He emphasizes the need for strategic innovations that enhance user experience and monetization strategies, particularly in the context of evolving AI capabilities.
"best possible UI to enable user to give feedback. And I think there's value in that. So, I think there's a certain amount of stickiness in there, right? So, what are all the different tools like in te..."
Patel offers advice on how OpenAI can enhance its value by integrating payment methods directly into ChatGPT. He suggests that by allowing users to input credit card information for transactions, OpenAI could capitalize on its capabilities in e-commerce, potentially generating significant revenue through a take rate model.
"Before we leave open, I want to ask a broad question which is if someone was was sitting here and saying, "Hey, Dylan, I'll listen to anything you you tell me to do. Any advice you have as long as it ..."
The discussion shifts to NVIDIA's impressive growth, with Patel analyzing the company's trajectory and the competitive landscape. He notes the accelerating demand for AI compute resources and the implications for NVIDIA's market position as other tech giants ramp up their investments in AI infrastructure.
"questions around this. I I want to shift to Nvidia. Uh Nvidia is having a monster year. They're up almost 70%. What are the possible paths from here? How do you how do you see it playing out? Um depen..."
Patel explores the economic factors influencing AI chip demand, highlighting the significant investments from companies like Meta and Google. He discusses the distribution of AI compute resources and the potential challenges faced by less economically viable providers in maintaining growth.
"third of it is like ads right um whether it be bite dance or uh Meta or many of the other people who are doing ads. So then it's still like okay well where are the rest of these one-third of the chips..."
In this segment, Patel addresses the disparity between the value generated by AI technologies and the ability of companies to capture that value. He argues that while AI is creating substantial economic value, companies like OpenAI are not adequately monetizing their innovations, leading to a broken value capture system.
"development right we know we can easily get about 15% more productivity out of a I don't think that's right I think it's way higher no no that we with a with a straight like I talked to a lot of enter..."
Patel shares insights on how automation through AI is enhancing productivity within organizations. He illustrates how his own company leverages AI to streamline operations and reduce costs, emphasizing the high value generated from minimal developer input.
"some like reality in that of course but you know ignores that like infrastructure spend today is accounting for five years of revenue not like one and the revenue looks like this not like flatline but..."
The conversation shifts to the potential for increased capital expenditure in AI infrastructure. Patel discusses the role of hyperscalers and sovereign wealth funds in driving investment, suggesting that there is significant untapped capital that could fuel further growth in the AI sector.
"source models like continuing to drive it down. It's like the value capture is just harder and harder and harder for these companies cuz they're making, you know, 50% gross margin on an inference if t..."
Patel examines the competitive threat posed to NVIDIA by custom silicon developed by major tech companies like Google and Amazon. He discusses the implications of these advancements for NVIDIA's market dominance and the potential for custom solutions to reshape the AI landscape.
"like like where it's not clear from, you know, if you have a spreadsheet, you know, and you're basing it on real business that you should actually spend this much. But people will because they believe..."
In this segment, Patel speculates on Google's potential to sell its TPUs externally, suggesting that such a move could significantly enhance the company's market value. He discusses the internal challenges Google would face in making this transition and the broader implications for the AI chip market.
"and there's all these um open source software libraries from you know Nvidia and and China and it makes the deployment costs like rock bottom then potentially hear me out here if if Google's TPU is is..."
Patel discusses the trend of commoditization in AI model serving, suggesting that companies focusing solely on serving models without developing them may struggle to maintain value. He highlights the importance of software libraries and the evolving landscape of AI, where open-source models are becoming increasingly competitive.
"for a long period of time. Um, and historically, no pun intended, software has eaten the world in most markets, right? I mean like if you look at uh early networking days Cisco was the most valuable c..."
The segment explores the influx of capital into silicon startups and the unique challenges they face. Patel notes that many startups have raised significant funding without launching chips, which is a departure from traditional practices. He discusses the competitive landscape and the difficulties new entrants encounter when trying to compete with established players like NVIDIA.
"right is that correct I think I talked to um one of the team members maybe maybe Rajco or someone about like why you guys don't didn't invest in like you know like a together or like a fireworks and s..."
Patel explains how hyperscalers like Google and Amazon can leverage their own infrastructure to compete with NVIDIA. He discusses the advantages they have as captive customers and how they can optimize supply chains to reduce costs, which presents a unique challenge for smaller silicon startups.
"which is you know why why right like why would you do this? Shifting gears what about the the silicon startups? Uh what's what's your take on those? I mean there's there's a ton of capital flowing in ..."
In this segment, Patel reflects on the historical context of new entrants in technology markets. He argues that successful newcomers often introduce disruptive technologies rather than merely improving existing ones. He discusses the importance of optimizing for specific workloads and the challenges of keeping pace with rapidly evolving AI models.
"that are like Yeah. Yeah. Yeah. That's fair. Um and then and then and then there's the old guard which continues to raise money, right? Like Grock and Surris and and Zanova and and Tenstor and so on a..."
Patel delves into the evolution of AI accelerators and the challenges faced by companies trying to optimize for transformer models. He discusses the trade-offs made by chip designers and how the rapid growth of model sizes can impact performance, emphasizing the need for continuous adaptation in the hardware landscape.
"effort in terms of team size. Um, all in the end, like, hey, I make a 75% gross margin as Nvidia. Um, AMD sells their GPUs for 50% gross margin. Um, and they have a hard time out engineering Nvidia an..."
This segment focuses on NVIDIA's competitive advantages in the AI hardware market. Patel highlights the company's ability to innovate and adapt its architecture to meet the demands of new AI models. He discusses the challenges faced by competitors in achieving similar levels of efficiency and performance.
"focusing on that to even prove out if it's worthwhile or not right um on a hardware side on a software side on a model side and so like you look at like Grocery Samanova um they all like sort of overi..."
Patel concludes by discussing the future of competition in the AI hardware space. He emphasizes the need for new entrants to achieve significant advantages over NVIDIA to succeed. The conversation touches on the complexities of supply chains, software development, and the evolving nature of AI workloads, underscoring the challenges ahead for competitors.
"constantly because of what because of what um works best on Nvidia and you see that with you know whether it be what DeepS's doing or Alibaba's doing or what the labs are doing internally um and you e..."
In this segment, the discussion revolves around the difficulties of creating a viable competitor to NVIDIA in the GPU market. The speakers emphasize that simply replicating NVIDIA's approach won't suffice; competitors must innovate significantly to gain traction. They highlight the importance of performance per watt and the challenges posed by the lengthy design cycles in the semiconductor industry.
"copper cables, everything. They're going to have better cost efficiency. So, you have to be like 5x better. But, but it to be fair, if if somebody had a viable competitor, which would even be marginal..."
The conversation shifts to China's unique position in the AI chip race, particularly regarding its power infrastructure. The speakers note that while China can support less powerful chips due to its abundant power supply, this creates a disparity in efficiency compared to Western companies. They discuss how this advantage impacts the global AI landscape and the challenges faced by American firms.
"shift, right? Because they're like, "Oh, what's the next generation of TPU and GPU look like? Okay, let's optimize for that." And the research path is, you know, like great, like yes, neuromorphic com..."
This segment delves into the geopolitical implications of AI chip production and distribution. The speakers discuss China's ambitions in AI and how the U.S. is responding to these challenges. They highlight the complexities of exporting technology and the competitive dynamics between U.S. and Chinese companies in the AI sector.
"because you have to be 5x there's a mode. uh because the supply chain stuff means that 5x actually turns into a 2 and a halfx and then Nvidia can compress their margin a little bit if you're actually ..."
The discussion focuses on the economic value derived from AI models versus the hardware that supports them. The speakers argue that while hardware is crucial, the real value lies in the models and services built on top of that hardware. They explore the implications of this perspective for companies like NVIDIA and the broader AI ecosystem.
"interesting, which is it's a big challenge in America, right? like um there have been there have been companies that were like they wouldn't they they've like you know Jensen keeps saying he couldn't ..."
In this segment, the speakers analyze how Chinese companies are navigating the global AI landscape. They discuss the strategies employed by firms like Alibaba and ByteDance to access superior GPUs outside of China and the implications of these actions for the U.S. tech industry. The conversation highlights the competitive pressures faced by American companies.
"moment they decide to um but there's these like there's like competing interests, right? Like because they want Huawei to be better than Nvidia. Yeah. And then this is how Nvidia argued to the adminis..."
The focus shifts to the capital investments required for AI infrastructure development. The speakers discuss how Chinese companies are increasing their capital expenditures at a faster rate than their U.S. counterparts, despite the latter spending more in absolute terms. They emphasize the importance of capital allocation in shaping the future of AI technology.
"Um I don't think so. I think I think again like there's a lot of like we what we see is that like even with H20 being sold to China in into China um and and and future versions of the chip the H20e an..."
This segment addresses the critical issues of power and cooling in data centers. The speakers highlight the challenges faced by U.S. companies in building infrastructure and the implications for AI development. They discuss the need for innovative solutions to meet the growing demands of AI workloads while ensuring efficient power usage.
"capital if they wanted to. um they're subsidizing the semiconductor industry to the tune of like $150 $200 billion a year uh through SOE through capex that's not generating revenue etc. So, it's not l..."
The conversation explores the future of data center infrastructure, particularly in relation to power sourcing and cooling technologies. The speakers discuss various strategies, including the potential for nuclear energy and innovative cooling solutions, to meet the demands of AI data centers. They emphasize the importance of addressing these challenges for sustainable growth in the AI sector.
"infrastructure really fast, right? Um and and their software is nice, I think, but like a lot of their customers are bare metal, right? Just just replace the GPUs whenever they're broken and network a..."
In this segment, the speakers discuss the broader challenges of building tech infrastructure in the U.S. They highlight the complexities of power grid interconnections and the labor market's impact on data center construction. The conversation underscores the difficulties faced by American companies in scaling their operations to meet AI demands.
"They need the power, right? And it's like um all the hyperscalers have like said screw off to my sustainability pledges because they need power as fast as possible, right? um they're, you know, they'r..."
The final segment focuses on the massive scale of investment required for AI technology development. The speakers discuss the financial commitments from major players like NVIDIA, Google, and Amazon, and how these investments are approaching nation-state levels. They explore the implications of this trend for the future of AI and the competitive landscape.
"data center and work on the wiring within the data center and all this other stuff uh the transmission stuff and your pay is up like 2x now uh versus what it was just a few years ago. This labor probl..."
In this segment, the discussion revolves around the cooling requirements of data centers and the misconceptions about AI's energy consumption. The speakers highlight that while AI data centers do require significant power, they are not the primary consumers of water compared to other industries. They explore innovative cooling solutions and the logistical challenges of implementing them effectively.
"at the end be every all data needs to be next to a nuclear reactor or lots of solar you know next to deep level like deep sea water that we use for cooling or something like that or what's um I think ..."
The conversation shifts to the economics of power in data centers, emphasizing that while power costs are rising, they still represent a smaller portion of the overall expenses compared to hardware and infrastructure. The speakers analyze the cost structure of GPU data centers, revealing that power and cooling only account for about 20% of total costs, which influences decision-making in data center operations.
"again like the cost of power like you go look at like these deals people are signing they're still signing like even though the price has skyrocketed uh from like a few cents a kilowatt hour for these..."
This segment focuses on Intel's position in the semiconductor industry, discussing its need for innovation and competitiveness against rivals like TSMC and Samsung. The speakers express concerns about Intel's ability to keep pace with advancements in chip technology and the implications of its current trajectory for the future of AI and computing.
"they were sitting idle, it's not worth it. Right. Just by like bypassing the grid, bypassing anything to do with interconnect, anything to do with public utilities. Exactly. Exactly. What's your take ..."
The discussion continues with a deep dive into Intel's operational challenges, including the lengthy product development cycle and the need for a strategic overhaul. The speakers suggest that while Intel has the potential to regain its competitive edge, it must streamline operations and focus on rapid innovation to avoid falling behind in the AI chip race.
"you want want them to be competitive? I think the process of splitting it would take so much executive time and so much executive effort that you would have been bankrupt by then, right? And that's th..."
In this segment, the speakers address the structural complexities within Intel, highlighting the need for a potential split between chip design and manufacturing. They discuss the cultural challenges that arise from this separation and the urgency for Intel to adapt quickly to avoid bankruptcy while maintaining its leadership in semiconductor technology.
"people or half the people working on it. And so like Lip Boutan to fix Intel needs to go into both the design company and lay off a shitload of people but like keep all the good people um and make sur..."
The segment concludes with strategic advice for NVIDIA's CEO, Jensen Huang. The speakers recommend leveraging NVIDIA's substantial cash reserves to invest in infrastructure and data center ecosystems. They argue that by expanding beyond chip production into broader infrastructure control, NVIDIA can enhance its competitive position in the rapidly evolving AI landscape.
"right? Uh which some could argue you need to lay off like 30% of the company anyways, but um there's a lot of bad things that happen if that happens, right? Um and they need to spend a lot more on bui..."
Dylan Patel advocates for NVIDIA to open up its TPU offerings and software, suggesting that a more aggressive approach could enhance their market position. He notes that the TPU team has been less aggressive due to talent loss to competitors like OpenAI, highlighting the need for NVIDIA to innovate and expand its product offerings.
"are you going to do with that? Um, I think I think there's something moving into the infrastructure layer much more um that they they could do if he really wants to be the king of the world, right? Uh..."
Patel discusses the urgency for tech giants like Google and Meta to adapt to the rapidly changing AI landscape. He points out that Google must address its inefficiencies and focus on integrating AI into its products to avoid losing market share, especially as competitors gain ground.
"like now I don't get as much in. I met some other people, right? But it's like you know um I think they could be a lot more aggressive in many ways across the company. They don't have to be, right? Bu..."
Dylan Patel critiques Apple's approach to AI, stressing the need for significant investment in infrastructure to remain competitive. He warns that without a proactive strategy, Apple risks falling behind as other companies innovate and integrate AI more effectively into their offerings.
"ship product better, right? Um Zuck, um I think I think Zuck, you know, it remains to be seen what goes on with super intelligence, but like they're trying to move super fast with the data centers. Uh..."
Patel highlights Microsoft's challenges in AI product development, noting that despite strong business relationships, the company is failing to deliver competitive products. He emphasizes the need for Microsoft to refocus on product innovation to capitalize on its market position.
"of their core IP every time they launch something is kind of mid, right? Um, you know, Metal Reality Labs is doing well, but I think they should like go more explicit. Like have a Chad GPT competitor,..."
Dylan Patel offers insights on what Elon Musk should prioritize at XAI, suggesting that while Musk attracts talent, he needs to focus on product development and avoid impulsive decisions that could hinder progress. Patel believes that refining product offerings will be crucial for Musk's success in the AI space.
"it, right? IDFA like they shut down ads to or data sharing to Meta, but Meta made better models and now they have way more data and way more power over the user than they ever did before kind of it wa..."
In wrapping up the discussion, Patel reflects on the key themes of the conversation, emphasizing the importance of infrastructure investment and product innovation for tech giants in the AI race. He expresses optimism about the future of AI while acknowledging the challenges that lie ahead for major players.
"company. Um, but like he's he's losing a lot of talent and and axing a lot of good projects. Uh, but Elon has a is a magnet to amazing talent and building stuff. So, I won't bet against him, but it se..."