0:05
Hello everyone.
Search this video
0:05
Welcome back to Semi analysis weekly.
0:06
We're here with episode 20 with the tokconomics team. Things are moving fast.
0:11
We are recording this on Wednesday, July 15th.
0:12
I assume by the time this episode comes out, there's going to be a lot more releases of models, disclosures from these companies about their profitability, revenue, uh how many active users they've got.
0:25
Regardless, we're gonna record a point in time right now talking about token budgeting, uh, Meta's comput strategy, MSL futures, the release of Fable, Soul, and Anthropics profit margins. So, excited to dig in.
0:38
With me today, we've got Max. How you doing, buddy? >> Doing great. Thanks for having me. >> Welcome. We got Crystal. How's it going, Crystal? >> Hey, Jordan. >> And we got Joey. >> Hey, Jordan. >> Cool.
0:53
So, we're going to start with token budgeting. Crystal, over to you.
0:56
We got uh lots of conversations that we've been h having with enterprises over time and just asking them like what's going on.
1:02
Uh at some analysis, we're still token maxing, but others are moving into the time of austerity.
1:06
Can you take me through a little bit about what you found with this article?
1:13
>> So, there's a lot of like uh enterprises right now that are cracking down on their budgeting because they think their employees are spending too much.
1:18
Yeah, a lot of people are getting scared because they're like, "Oh, what if I don't have enough tokens to do my work?"
1:22
I don't think that's necessarily true then, and it applies to most people, right?
1:27
Because a lot of people aren't even getting close to that limit.
1:28
There's probably a few handful power users at each of these companies which will really be affected.
1:34
But other than that, I think it's been it hasn't been as big of an impact, I think, on most people's workflows as it has been portrayed on social media and stuff.
1:43
There's been a lot of different strategies that companies have been imposing whether it's like on a per person basis or like a monthly budget for the entire company from what we've seen.
1:53
And I think the per company basis obviously makes a lot more sense than like on a per person basis just given that some users will generate more like use from it than others.
2:05
>> And how do you see people actually enforcing this?
2:06
Like clearly you can burn through a budget faster if you're going with the super ultra premium max thinking model, but some in some cases when you're actually enforcing a budget, you take different approaches, right? >> Yeah. Yeah.
2:18
So it I've seen some people actually so I was talking to some people and they are like a smaller company and they have a much smaller budget.
2:27
So the way that they do it is they try to optimize how they're using the like more expensive tokens for like anthropic or open AI.
2:35
So they use like a cheaper model to like process a lot of it and like condense it down to like a smaller prompt and then they use the more expensive models or they use something that their company's not counting.
2:44
So there's a lot of like discrepancies in terms of what models actually count.
2:49
Some people or like some enterprises are counting their own models into the budget but some enterprises aren't.
2:53
So there's like if they aren't counting it, a lot of people will tend to like really push those to the limit, use those a lot and then use like the ones that actually cost money.
3:05
And you're seeing people maybe Max I can bring you in on this one.
3:07
You you've been seeing a lot more people burning tokens for coding rather than other use cases, right?
3:14
Like that seems to be the market that's growing the fastest.
3:18
>> Yeah, I think coding/ like software jury in general is just by far the most token hungry use case.
3:23
And I actually think this is why a lot of the token budgeting discourse is pretty like I think a lot of the budgets themselves are pretty uninformed.
3:32
Like I hear people saying that they want to make sure their sales guys aren't using Opus or Fable to write emails.
3:38
So they should definitely be using Sonnet instead.
3:39
And it's like dude like generating your email is basically free from a tokens perspective.
3:44
Like it does not matter if you use Opus or Sonnet to write this email.
3:48
Like I have no idea why this is the policy you're enforcing to try to like reduce your token spend.
3:54
I've also heard lots of stories of like certain companies where only the engineering team is allowed to use like cloud code or codeex and then you know everyone else only gets like codework or something like that.
4:03
I think that's pretty shortsighted and it's honestly you shouldn't have a cast system where your engineers are at the top and everyone else is just forced to use like demoris.
4:12
You should really be giving everyone the opportunity to try these tools and figure out how you can become more productive with them. >> Yeah.
4:19
Joy, are you using these models right now?
4:21
How would you feel if I got access to it and you didn't?
4:24
>> I would probably be I do use yeah some access not as much as like Max and other people on at semi analysis but that's probably just the nature of our work or my work right now and that's changing a little bit but yeah I'd be I'd be pretty pissed because I still use Opus 4.
4:37
8 a lot like to pull in to do more like aic research tasks that will pull in a lot of like different data sources do more like research type tasks.
4:46
Um yeah be I'd be pretty mad.
4:49
>> How do you think about the ROI question?
4:51
And I think a lot of people the the whole motivation behind token budgeting is clearly that somebody is not seeing a return on this investment.
4:56
Either that or they're just like not seeing it yet.
5:00
But I think in a lot of that research you're doing, you're seeing that there's a lot of benefits, right, to using >> Yeah, I think like with the market being so coding focused today, so probably 70% plus of AR lab AR right now on the API side is in coding.
5:16
And then two, like from what we've heard, it's very power user a power company driven.
5:19
So it's pretty widespread like coding.
5:21
There's no like major big company for anthropic.
5:23
Like meta is probably 3 to 5% of total anthropic ARR but it's heavily driven by power users.
5:32
Um if you're at a point where you're a power user at one of these companies I think the ramp data is like the top 1% you know the 99th percentile of companies they spend 100,000 you know per year on AI per employee.
5:41
And if you're if you've gotten to that point like where you're spending that much you're obviously getting pretty good ROI.
5:48
And I think like our conversations with a lot of enterprises were these top engineers could go, you know, well above, you know, their their limited bud, you know, whatever their budget limit was.
5:58
But if you were someone who was spending, you know, an insane amount of money, you know, you were getting budgeted already pretty quickly if you weren't seeing ROI.
6:06
Um, and I think, you know, there's always projects that you'll try that don't get you'll scrape that you'll scrap that and then move on.
6:14
But I think it was yeah it was pretty obvious that because the market is so coding focused like people are getting ROI they continue to spend.
6:20
I think we continue to hear then net new AR both anthropic and now open AAI uh with like codeex and in 5. 5 5.
6:27
6 like since um late March spend is broadening there's clearly a lot of like API token spent today.
6:42
Yeah, there's a lot of organizations like in the, you know, that aren't traditionally, you know, doing a ton of, you know, development type work or focused and they're, you know, they only allow their like middle back office workers to use, you know, Copilot 365 or SAT, you know, things like that.
6:56
Um, but I think the market will continue to broaden out.
7:00
Um, >> do do you guys did you guys have a take on the balance between API usage in applications versus like coding plans when it comes to people's individual usage?
7:14
Like do you ask them when you're talking to them if they are using a cloud code subscription versus API paying per token?
7:24
>> Well, I mean anybody who can use a subscription definitely should use a subscription just because it's so subsidized.
7:29
It's just you know OpenAI and Dropic are very wise to this and so when they have their enterprise plans like same analysis is on a you know a drop enterprise plan they just they don't let you use a subscription they force you to pay for a token because they know that's where the margin is but I think in general like anyone who can use subscription does use subscription and it's just the people who need more limits than that that pay for API.
7:51
>> I've even talked to like a few people at like startups and stuff and they're not on like the team or the enterprise plan at all.
7:56
they just buy five, however many like uh Claude or OpenAI subscriptions they need a month, charge it to the company card, and they just keep doing that.
8:05
If they hit the limit on however many accounts they already have, they just buy another one for the month because it's just more cost effective that way for them to operate than it is for them to actually use the API billing.
8:14
But yeah, if you're a startup that you're like young enough to where you don't care about like security or any of the other enterprise guarantees, like just absolutely buy, you know, 10 subscriptions per person, >> you know, paying for API credits.
8:27
Like I I think the recent uh, you know, study we did on this, it had a decently viral tweet said that >> for Codeex, I think $200 a month, you get like around 12K worth API credits and for anropic it was like 200 a month for around 8K.
8:41
Like obviously that's a no-brainer for one that is able to pick between those two.
8:49
>> Can you dig into that a little bit more, Max?
8:51
Like when you're talking about the margin profile of these businesses and then you're considering the subscription plan versus the API like what is that uh what does that look like?
9:01
I'm going to bring up the chart on screen that I think is the key one from uh that tweet thread. >> Yeah.
9:08
[clears throat] Um, yeah.
9:08
So, essentially, um, obviously when you hear that like a $200 a month plan can generate $8,000 worth of, uh, API credits, uh, the subscription is going to be much lower margin if the average user uh, has anywhere near 100% utilization.
9:27
Um, and so really the key question is sort of like what is the actual average utilization across our entire user base?
9:34
And this chart that Jordan showed us here is like kind of the break even percentage um for the various plans.
9:38
We can see for the Cloud Max 20x plan, it's like just 10%.
9:44
Um for the Pro and 5X, it's like a little better at 20%.
9:47
Uh OpenAI because they are sort of even more generous with their subscription limits is slightly lower at 11.
9:53
4% for the Plus and Pro 5X and just 5. 7% for the Pro 20X.
9:59
Um we're pretty confident that the average utilization especially for these like 20x plans is much higher than 10% and uh 5.
10:08
7% respectively for uh anthropic and openi uh which would mean that like forget being you know worse margin than API which we think is like likely 85% plus for enthropic at least uh it might just be like negative margin in general for these 20x plans and so of course like these businesses want to move as much volume as possible to API.
10:32
Um, and I think maybe Joey since she wrote the newsletter on Thropics business, this would be a good segue to talk about why we think Anthropic is potentially in a stronger position than OpenAI. >> Yeah.
10:42
And and I think too like once you move on to enterprise um subscription plans at Anthropic, there's no usage included in your subscription fee.
10:50
It's all it's all done at API pricing.
10:54
Yeah, I think at Enthropic what we found was really interesting is because the business you know today is so much more and this is this is changing but because it's you know 80% plus of ARR is on the API side and that is as Max showed is is really high margin
11:12
you know they've gotten to a point where they're profitable today um you know and some of that is on an on gap basis so excluding stockbased compensation but like operating profit um was was profitable in Q2, you know, could be profitable to the tune of 1 billion plus in Q3. Um, some of that's a function of
11:28
Um, some of that's a function of they've just grown so much I don't think they can hire enough or like plow enough back into training as as they'd like.
11:37
But because of that margin, that kind of gross profit advantage that they have today, um, they can plow as much back into training as they want um, and can kind of extend their model advantage and model capability advantage.
11:48
We think that's something they should obviously try to do.
11:52
So just because it's, you know, very profitable for them today, you doesn't mean they should go run at, you know, 30% ebid margins right away or as quick as possible, show that this is a very profitable business model, but continue to invest as much as you can into training as possible. >> Yeah.
12:07
Can you can you talk about the mix between consumer and enterprise?
12:11
Assuming that consumer is those paid plans and enterprise is pay peruse on the API like >> Yeah.
12:16
When you when you look at these things, clearly the bulk of like Twitter users are using the subscription plans, but then you get like one really, you know, big customer who's spending so much more money on the API tokens that it kind of drowns out everything else and that I guess makes it a good business. >> Yeah.
12:36
And because Anthropics really focused, they have been historically focused on enterprise.
12:39
It was 90, you know, it's 90% plus enterprise.
12:46
you know versus chat GBT has gone into you know started heavy in consumer up in you know back in Q1 60% of AR was in consumer and only 6% of the 950 million weekly active users end up actually paying for a plan and most people pay
13:02
for the $20 a month plan it's a much different business mix because then you know OpenAI has to subsidize and pay for you not subsidize but there's a cost to serving those other you know 900 million weekly active users that are free. So
13:14
So lowers their gross margins on a blended basis by about 20 points.
13:22
You know, that's changing.
13:22
So I think, you know, we think personally with some of the rise in OpenAI's API business uh since late March with Codeex and then 55 and 56 um that the 6040 consumer enterprise split, you know, has flipped now to 40, you know, 60 consumer versus enterprise here in Q2.
13:40
And it's, you know, from what we're hearing like in the channel, um, especially out of, you know, token as a service businesses like Bedrock and Foundry, um, is that, you know, Open AI has recognized this and it's starting to shift and even, you know, AR like net new AR is starting, you know, to look a little more even between anthropic and open AI. >> Yeah.
14:01
And it's kind of shocking like with the release of 5. 6 six soul.
14:02
The guys like Tibo on X who are tweeting about the daily actor user counts are saying that they're celebrating going to seven and then to 8 million active users.
14:17
But to be clear, they've got over 900 million weekly active users on the free tier of Chachu BT.
14:23
Like they've got a lot of room to grow to just get people using the number one paid product.
14:30
Maybe they got a few extra users using other paid products, but I would assume the bulk of people using the paid products are using codecs, right?
14:39
So that's not a great conversion rate at this point of in terms of who's actually using the full paid product from the, you know, base of 900 million. >> Yeah, it's funny.
14:52
You actually see it too in the paying rates for like even the consumer version of of Claude.
14:56
like cloud is like 50 like 9% of free users end up paying.
15:02
So it's a much more like focused user base.
15:04
I think you if we think of those 900 million weekly active users a lot of it's like Google search type replacement things like that.
15:10
And then you know obviously because of that consumer focus they've been working on advertising a lot consumers traditionally monetized you know people aren't willing to pay the retention curves are typically pretty poor.
15:21
um and how you know people how many people are are paying you know 3 months later, 6 months later, 12 months later.
15:31
Ads is a tough business model.
15:31
It's tough to tough to scale but you know maybe that's the eventual you they've put out some pretty crazy targets for 2030 you know advertising revenue but yeah it's consumer is a tough business enterprise is is you know obviously given the margins on tokens for API like enterprise is the place to be.
15:52
I think Anthropic had a pretty good lead because they're so lever decoding early, you know, the first six months of this year and it seems like it's shifting to more of a kind of equal two horse race uh here in mid July. >> Yeah.
16:04
Crystal, what are you seeing in that two- horse race?
16:05
Like are people uh that you're talking to using anthropic codecs both?
16:14
>> I feel like a few weeks ago when I was talking to people, people were pretty heavy on using anthropic cloud code specifically.
16:21
And if I asked them why, they would say that's just what I've been using, right?
16:24
And I'm used to using it now.
16:26
But now as people in like the consumer section start using like chatbt a lot more than like claw, that effect in like the consumer sector kind of starts trickling into like enterprise a little bit where like these people who were previously like claude code like power users didn't touch codecs at all.
16:45
They're kind of like okay, what if I like consider tinkering with codec?
16:46
So even though consumer is like a really small portion of the pie and the margins obviously aren't as straight as enterprise like and historically speaking open AI hasn't done as well in the consumer market as like Anthropic has but like they have the that traction in the consumer space that eventually like is going to trickle in and we're already kind of starting to see that in like just this past week already.
17:13
>> Max, what's your take? OpenAI versus anthropic.
17:15
Where do things stand for you?
17:18
Uh I think I remember last time I was on this podcast, we were talking about how it was it was right when 55 came out and we were talking about how things uh were looking really dire for open eye at the start of the year.
17:27
Um I think ARR growth was likely flat in like March and April which is the alarm bars are definitely ringing, you know, in Sam's head when that happened.
17:35
But then we said that 55 would like to be an inflection point.
17:38
Like this was actually a really good model, you know, probably opus 48 level.
17:42
And I think that prediction has played out even like sort of more quickly and more optimistically than we would have guessed.
17:49
Like some people here are saying that 56 is as good as Fable sort of just like straight up even though it's only half the price.
17:56
And we also likely think it's like a much smaller model which would be sort of a testament to OpenAI's guess like training abilities.
18:03
We think that like as Joey mentioned earlier, net new AR like month on month is like a comparable between open AI and anthropic now which is pretty shocking like people people were counting OpenAI out a little bit and I think Jordan might have been one of those people but [laughter] they're but they're back guys they're back.
18:23
>> I am nothing if not flexible.
18:23
It's always been about the model guys like everybody's like when are you gonna stop hating on OpenAI as soon as they ship a model that's good and it's good.
18:30
I mean, for the things that I want to do, yeah, I'm Yeah, basically default OpenAI at this point, despite the application not doing what I want it to do with multiple tabs and stuff like that.
18:45
>> Wait, are you using the Codeex app or like the CLI? >> CLI.
18:48
>> What's your form factor? CLI.
18:48
I feel like they don't want you to use the CLI, dude.
18:52
Like they they very actively want you to use the app.
18:53
What is uh what is pulling you back? >> Tabs.
18:56
I need like multiple tabs going. Yeah. Uh, okay.
19:01
I think I think the way you're supposed to do this because I agree like the first time I tried using codeex and I was like every single time I open a new chat, it's just like lost in the ether on like, you know, the lefth hand sidebar of all those tabs.
19:11
I think how you're supposed to do it is like you have one pin thread like per project you're working on and then you just like ask that thread to spin up subthreads anytime you have like a discrete ask.
19:25
And so you actually like never even interact with the vast majority of threads like in the left sidebar.
19:28
It's just like your couple pen threads that you use. >> Yeah, that sucks.
19:32
Second of all, >> second of all, I'm in remote SSHs all the time and their remote in the thing sucks.
19:45
So, yeah, both of those are I'm very against it. >> Makes sense.
19:50
>> I think they are working on this, but yeah, I'm I'm just I'm not reading the code to be clear.
19:57
>> Thanks for clarifying.
19:59
just yeah last time you were on I you know we had that discussion but you know still have to run some things still have to run some things from the terminal and see some output in the logs at some point and this is just like not part of the standard codeex thing but I'm not
20:13
the primary market like I understand that they are targeting vibe coders around the world you know >> we need to help people make more B2B SAS is like the core business model of this and that's I'm not the target market and that's cool I'll keep using the CLI. It's all good. It's all good.
20:31
>> Actually, I wonder if like internally most of their software engineers are using the app with CLI.
20:34
It's semi analysis for sure.
20:36
I think all of our real engineers are like CLI, you know, die cards, right?
20:39
Like they'll never give it up.
20:41
>> That would be a good question.
20:41
You're going to ask them that. >> Yeah. >> Yeah.
20:45
>> Well, next time we I see some open people, I'll ask. >> Sounds good. >> Yeah.
20:50
>> Okay, let's uh let's change gears and talk a little bit about who's going to come in third.
20:53
There's been some releases uh that we've covered a little bit. SpaceX AI is back.
20:57
They've got Grock with cursor bolted on and things are going well.
21:03
Meta has mus spark out in the world.
21:06
They've also stacked up a whole bunch of compute and they've teased launching a Neocloud, which of course M SpaceX was first to do.
21:12
So, what do you guys want to talk about first, the models or the compute for these guys? >> Let's do compute.
21:25
I think it's more interesting.
21:26
NASA is going to tell you to walk through the models.
21:27
I think yeah, >> I mean I think quip tlddr the models is like they're not bad but at this point I don't think it matters unless you can deliver like a true frontier model and they're obviously not true frontier so who cares.
21:41
>> Do you think anyone will >> max their API like product?
21:47
>> I don't think anyone's going to use the API until it's a frontier model. >> Okay. >> Yeah. Yeah.
21:51
>> So when they when they have an API, their token as a service business will be >> will be fire% margins too.
21:58
And >> I mean that's only if it's like better than whatever the best entropic and OPI model is at that time which I think easier said than done.
22:07
I think like Grock 45 spark 1.
22:09
1 they're only useful as like uh just like proving points along the scaling curve.
22:14
It's like okay these guys unlike Gemini they're like still like kind of in the game right now.
22:20
um you know they can train like adequate models.
22:22
It's not quite frontier yet, but there is a nonzero chance they can catch up I think is the way to view them.
22:28
I will personally be shifting a zero with my token usage to either Grock 45 or 1. 1. >> Okay.
22:37
But you said the other name which is shockingly in my view in fifth place right now.
22:43
Gemini, do you still that view? >> Oh 100%.
22:46
I think they're like clearly in fifth place.
22:48
[laughter] And like unless unless Gemini 3.
22:51
5 Pro is like better than the you know industry chatter that we're hearing, I think they're going to stay in fifth place maybe forever. >> Maybe forever. Okay.
23:01
Talk to me about compute.
23:04
So staying in fifth place forever would kind of be related to the quality of the model, but the quality of the model is kind of tied to how much compute these guys have. >> Yeah. Yeah. Yeah.
23:14
>> Yeah. Yeah. Yeah. And I think so I have to say that when I first saw the SpaceX entropic deal where Elvon rented like basically all of Colossus one to Atropic, I think I incorrectly viewed that as Evon when in reality like I think the really important question is like do you have the ability to claw back all of your compute assuming that you have proven that you can truly reach the frontier by scaling up your compute
23:41
and I think that's sort of like the view that uh Elon is taking with like uh cursor and XAI is essentially saying like I'm going to leave you guys with enough compute to sort of prove that you're capable of reaching the frontier and in the meantime I'm going to monetize all the additional compute by like renting it out on these like extremely high margin you know 3x maybe 4x market rate uh rental deals. I'm
24:03
I'm going to make sure there's this like 90-day clawback clause included in every single one of them such that if you do really well, I'm just going to claw back all the compute, give it to you so you can, you know, make digital costs.
24:14
And I think that is like a valid strategy for someone who still truly thinks they can like build RSI.
24:22
And I think you can really think of where it's like OpenAI and Enthropic are monetizing their compute by just like selling inference, Elon is uh doing these short-term comput deals instead. I think this is fine.
24:35
I think on the topic of meta compute like you know there are the rumors that they're going to become a neocloud.
24:39
I think as long as they do these like SpaceX style deals the clawback or they do like or they just like give it all to Rexus or they like do token as a service like any monetization method that allows them to in theory claw it all back if MSL proves they're doing well.
24:55
I think that's like an excellent smart decision on men's part.
24:59
Probably allows them to be more aggressive with buying compute in the short term and it still like gives them a chance to build RSI in the long term.
25:05
I think the issue with Google and why they're in fifth place aside from their model like Gemini 35 Spy Flash being bad is that they have signed all these long-term compute deals where they cannot claw back the TPUs that they're giving to right that's just like committed for the long term and I think it indicates like a lack of conviction
25:23
on their part that they think they can build RSI and I think you know Nom Shazir John Jumper like all these really high-profile guys at deep mind leaving is sort of because of this fact like they are hi pill their leadership is not hgi pill and they think that the only way they can do the only they can do in this situation is to go work somewhere else. >> Wow. Okay. Hot takes. So the two things >> Wow. Okay. Hot takes.
25:44
So the two things I want to dig in on on there.
25:49
The first is related to the clawback.
25:49
So this is a valid strategy but you can basically only do the clawback once and then nobody's ever going to buy from you and trust you again.
25:58
Or do you think you can actually between these two?
26:04
Well, I think in an ideal world, you only need to do it once because you only do it if you're confident that like I have I'm going to be able to breach the frontier by continuing like scaling, right?
26:14
Uh but I also think it's possible you could do it more than once.
26:16
I think in principle, Enthropic is like viewing this as a short-term deal.
26:19
And I think if Elon said like, "Hey, actually, I want it all back six months from now and then a year from now he's like, JK, uh we actually still can't train a frontier model."
26:30
If enthropic is so ripping at that point and it's in desperate need of like computer capacity, why would they say no?
26:37
>> Yeah, I guess financial motivations can make friends out of enemies.
26:40
So second thing is just on the topic of the business maybe Joey I can bring you in here.
26:49
What's a better business?
26:49
Selling tokens of your frontier model or close to frontier model or selling compute access on the short term at a premium over the market?
26:59
Selling your own tokens is probably the best model.
27:01
I mean, even like even selling someone else's tokens if you're AWS or or Azure is a pretty good business model as well.
27:07
If you can participate in some of the the upside revenue share, but if you can run above, you know, X market rates, it's not a bad business.
27:15
I think before like in like from the MetaMP compute perspective, it's been floated for a while.
27:22
You met compute has been floated since like Q3 of last year.
27:24
And it gives Zuck a backs stop to say, okay, if MSL isn't successful, we have means to essentially make decent ROI on our capex spend.
27:36
It allows them now to spend a ton more on capex in 27, you know, potentially 28 because there's a market and, you know, you've gotten these market signals that Anthropic can pay, you know, 3x market rates and, you know, you can do a small deal here and that's in, you know, the background.
27:49
investors can feel comfortable that you know if um if MSL is is deemed you know unsuccessful that you know Meta could have a pretty nice you know single client or multi you know few client like Neocloud business just serving the labs. >> Yeah.
28:06
But I mean I I thought this chart that you guys had in the MetaMP compute article was striking where you can see that the SpaceX plus Google deal they're monetizing this compute at 4x the market rate.
28:17
So, I think in the past everything you're saying makes perfect sense, but I would challenge a little bit that like this near-term SpaceX GB300 rad deals aren't a really good business.
28:34
Like they're clearly high margin in this, right? >> Yeah.
28:38
I mean, that's extremely high margin.
28:39
margin. I think like when you think about the you think about like the AR per per like gigawatt I think a lot of people be happy to get 48 billion of like actual token AR like of inference there are so I don't know that deal
28:50
doesn't seem very economical for for Google but you know >> telling a you think their meta compute is them telling a story to the market that they're going to be able to replicate this space XAI thing which is probably a one-time thing unless demand
29:04
keeps going so exponential that there's such a compute crunch that everybody's fighting for access to the GPU And while we have them and we can monetize them at like 4x the market rate from Neoclouds where Neo clouds I think we already I think our house view is that Neocloud is
29:19
a pretty good business on its own at like 12 billion per gigawatt approaching 50 billion per gigawatts like >> yeah and like >> and the labs have gotten so good at the AR per gawatt or gigawatt like number has gone up a ton. So token throughput's
29:34
So token throughput's been pretty impressive like just you know the models as well and then two how they've monetized them you the new models price higher.
29:41
So like if you look at like anthropic like revenue per megawatt was like 16 million per megawatt last year or 16 you know billion per gigawatt and now you know that's that's more than doubled in in three quarters and I think there's some people that think that can double again in two to three quarters.
29:56
So >> let's look on that trend.
29:59
I've got this chart on screen right now.
30:00
So, uh, total token as a service market, I two things jumped out at me.
30:04
One, how fast this thing is growing.
30:07
Like, I thought the market was pretty big in Q4 of last year.
30:11
And that looks puny [laughter] compared to our forecast that you guys have on this chart towards the end of this year.
30:19
Um, like way more than doubling as you said.
30:22
Uh but the other thing that strikes out that jumps out to me is just how bad Microsoft foundry is doing here.
30:30
Can you >> Yeah, and that'll be revised up with the recent OpenAI and API success.
30:38
But I think up until because open you know they were you know foundry was 90% OI and you know OAI was so consumer focused and and API didn't really this is not the the latest version of this chart.
30:50
Um yeah Foundry is you know doing a little better now. Maybe Vert.
30:52
Ex is doing a little worse given, you know, Matt's or Max's thoughts on on Gemini >> than that chart.
30:58
But >> you got to subscribe for the most recent tokconomics forecast.
31:02
These companies always we can't be giving this away for, you know, for free, dude.
31:09
>> You got to you got to pay for the tokconomics.
31:11
>> We were nice enough to leave the axis on that chart this time.
31:15
>> But yeah, I think we're like we're of the view and like when we talk to enterprises, you know, token as a service is a very popular, you know, choice for consuming tokens.
31:25
You know, AWS and and Azure have massive businesses with Fortune 500, global 2000 companies that, you know, spend nine figures plus, you know, on on cloud a year.
31:34
And if I can go to an existing vendor and have more model optionality, increase my ELA or my my credits um and then burn that down.
31:45
It's it's pretty pretty attractive um for both AWS and Azure.
31:51
And I think like you know if it was 5% indirect like sources like you know this token as a service for like anthropic 6 months ago like that's probably it's probably 20% now of your business.
32:04
>> Crystal what's your take on people using the big three hyperscalers token as a service stuff versus the smaller startups like a together base 10 fireworks anything they can get on open router.
32:19
I feel like as we see more of the bigger enterprises, so like I feel like a lot of the financial services industry still hasn't unlocked like and use AI to the level that they probably can and should be.
32:31
And that's where like Azure and Bedrock are going to be benefiting from because most of them probably already like buy their services, right?
32:39
And then once they do get like internal approval or whatever to be using these AI tools, like they'll just buy it through whatever existing channel they'll have already, whereas it's like probably going to be like that's where the bulk of the market pretty much is. >> Makes sense. Yeah. Max, how about you?
32:58
What do you think of token as a service from like the startups?
33:02
how fast together Fireworks base 10 they're growing compared to the hyperscalers that are kind of tied to the big labs but maybe not actually selling a lot of tokens to the startups of tomorrow.
33:17
>> Yeah, I mean I think uh together base and fireworks these are all super impressive businesses.
33:21
I do actually expect like opensource token volumes to grow slower than frontier token volumes, but that's only because I think frontier token volumes going to like absolutely explode.
33:34
I think open source token volumes are also going to explode just to a slightly smaller degree.
33:37
And I think like it's kind of funny.
33:39
I feel like if you talk to the VCs investing in like together fireworks and base 10 and you ask them for the explanation why oftent times it lead with like infrance is going to be like the largest market ever.
33:50
And so it doesn't even matter if like these companies can only capture like a super super tiny slice of this like extremely large market.
33:56
That's good enough for us.
33:58
And I think that thesis is like honestly more or less right.
34:00
I don't expect these guys to ever be doing like more volume than an open eye or anthropic or like nowhere near that.
34:08
I I think like this is in terms of like global token volume.
34:11
They might be like the highest percentage they've ever been like they ever will be rather today.
34:15
But they'll still be like good businesses in the future, I think. >> Yeah.
34:20
How about the rumors that they can hitch their wagon to some of the bigger labs?
34:24
So like SpaceX AI starts to win a little bit and then Fireworks grows because they're exposed to Cursor and actually have some I don't know exposure to that growth. >> Yeah.
34:38
Well, I don't think that's going to like last long term.
34:40
I think the only reason why fireworks got that exposure is because originally composer was post train on chem, right?
34:47
I think if you're a frontier lab that has made like a new model from scratch, there's no reason to believe why fireworks engineers would be better at optimizing that model than your own engineers.
34:58
So I don't think they can get like any share of that margin in the future. >> Okay.
35:05
So why do the hyperscalers get a share of that margin with the big three?
35:11
I mean because they just have like and I'm sure Joe can speak more on this uh but they just have like the enterprise distribution that the neoclouds do like Enthropic is not giving up this margin to bedrock so that Amazon engineers can
35:23
like make claw run faster and you know B300s or whatever or >> it's just so >> yeah sure trium but uh it's just so like uh all the existing customers that are reliant on bedrock uh can also use cloud malls >> sense yeah >> yeah I think the deal. I mean there's we
35:39
>> yeah I think the deal. I mean there's we hear more we don't hear anything but people tell us you know they think the deal could get reerruck um because it's obviously you know it was very beneficial for AWS I think we've written on AWS and their margins and how they
35:53
monetize you know versus just selling the bare you know compute for anthrop to run inference on you know to get that 30% you know or 20 to 30% revenue share you know that just falls down to the bottom line is pretty attractive AWS like Azure lesser extent GCP like obviously have massive customer bases
36:13
people's cloud estates are there the enterprises are very comfortable security and compliance and you know everything now with cloud even your more regulated industries are so it's a natural place for them to want to buy but to see that much of the economics for anthropic is obviously yeah I think
36:29
a lot of that was because when the deal was was struck and kind of the incentive to make tranium work on that you know they were able to to run with those terms and make it pretty favorable So yeah, we'll see what happens if those revenue share deals can continue. >> Okay, let me make one more attempt at
36:44
>> Okay, let me make one more attempt at the bullc case for these token as a service companies that aren't the hyperscalers for AWS and Google.
36:52
Obviously, a big portion of the benefit of getting token as a service from them is that their engineers are are working on tranium and TPU and that might be different than the experience that the lab has with the GPUs where they have more experience like running this themselves.
37:04
There's a whole class of chip startups that are coming to market right now and an obvious way in which they come to market is by partnering with these token as a service companies where hey the the biggest Frontier Labs aren't going to spend a whole bunch of time optimizing for you know the seventh best chip startup that's coming to market right now.
37:24
But if they do and that chip startup strikes something really nice for a given model that makes it a lot more cost- effective or a lot higher performance to run instead of Nvidia, then they've got a shot at doing something super unique.
37:35
Do you think that's a, you know, potential future in terms of where these kernel engineers who have learned a lot end up going and spending their time over the next few months or years?
37:51
This is a good point and it's reasonable in the short term, but if any of these new chip startups actually reach like sufficient scale, I think the labs will just dedicate teams to making their model like run really good on their chip.
38:05
I think there's just I think there's no world in which you have a new accelerator that's like meaningfully better than Nvidia and it's also being like uh sold at large volume that OpenAI is like relying on together to serve their model like on that chip instead of just like working directly with that company to you know develop the first party capabilities to run their model on that chip. >> Yeah.
38:29
You think it's going to go the way of Cerebrus where the chip companies to be successful are effectively going to have to become a neo cloud for themselves and exactly relationship with the frontier labs. >> Okay, makes sense.
38:39
Maybe we can finish by talking a little bit about MSL with Meta particularly Max.
38:43
I just loved the like crash course on RL that this article turned into.
38:47
not necessarily how it works from a technical perspective, but how it works from a business perspective like where people buy data, how they build these environments, what sort what the market looks like.
38:59
Can you kind of give a an over like previously in the world where everything's pre-training, it's just like whoever has a, you know, frontier class team and the most compute can just train the biggest model and win.
39:14
They follow the scaling laws and go there.
39:17
But now there's a scaling law related to RL.
39:19
How how is this playing out in your mind at a high level? >> Yeah. Yeah.
39:26
I think it's I think a it's like really important for everyone to understand that reinforcement learning or RL is like probably the most important scaling law for improving model capabilities today.
39:37
And there are a lot of people who believe that the only thing that's stopping the models from being able to do like literally anything a human can do on a computer is having sufficient RL environments.
39:46
This is sort of like data that lets the model like try to complete white collar tasks itself until and it can sort of like repeatedly try cleaning the task until it fully learns how to solve it.
40:00
And there's like an entire new industry slash like supply chain of these RO environment startups whose like entire job is to convert real world economically viable tasks into these RL environments that can then sell to the labs and allows use it to improve their models.
40:18
Um yeah, you see like most of the main players um uh on this chart Jordan has pulled up here.
40:25
Uh I actually think like some of these AR numbers might be slightly understated.
40:29
Uh I think it's like pretty consensus that uh the total sort of data budgets at the frontier labs this year.
40:36
So this is like um primarily OpenAI, Anthropic, Google, Meta and XAI, but also the longtail companies like uh you know Meta uh sorry Amazon, Microsoft like Thinking Machines.
40:49
Um I think all those sum together is going to be like well over 10 billion.
40:53
Um that's sort of like a 10x relative to last year.
40:55
It's very possible with 10xes again um in 27.
40:57
uh and this is just like a hugely important market for improving uh AI capabilities.
41:07
Uh this is probably like maybe the only market in the world where in customer demand I guess maybe compute was the other one but like in customer demand is just not even a question for all these startups like they will never have a contract turned down from the labs because it's too expensive.
41:23
It's just a question of like can you scale up you know creating high quality data fast enough and if the answer is yes we will pay any price for it.
41:31
So yeah this is like I guess maybe as one other like side note one of the reasons why Enthropics models are the best at coding today um or at least like they definitely were before 56 soul came out is that they were by far the most aggressive from buying coding data from all these oral environment startups.
41:47
Um I think some of the other labs are starting to like realize this and catch on.
41:52
Uh but it is definitely like a super important industry everyone should be.
41:58
>> Can you dig into a little bit of the process to create some of these tasks?
42:02
you kind of ran through this in the article by describing how Meta has moved thousands of engineers doing this work and uh and also dispelled the notion that this work is like meaningless soul crushing stuff and is actually you know pretty both economically valuable and I don't want to steal your thunder but you had a nice line on that one.
42:24
I think I said it was both potentially more economically valuable and intellectually stimulating than like your average big tech job.
42:30
So like yeah, I think a lot of people think like they hear the the phrase AI data and they still think like oh we have some random people in the Philippines who are like drawing bounty boxes or like uh labeling text as like NSFW or something.
42:45
And that's just like like dude, the models have already fully solved that.
42:48
Uh like your data is only valuable if it's uh doing something that the models don't already know how to do.
42:56
And so what this means in the case of software engineering is that in order to make a good like software engineering task today, it typically needs to be something that would take like a really good human engineer, maybe like a full day of work to solve in order to be something that like the model can't already run on itself today.
43:13
And so if you want to create this data like essentially you need like a really good human engineer to sit down and like think of an example problem that he would actually want to do.
43:24
You then need him to like create a verifier like usually a set of integration tests maybe along with the rubric that can check if the model actually susly completed this like dayong task or not.
43:35
Obviously this is easier said than done.
43:37
is easier said than done. And then you also need this engineer to like write a prompt for the model that like asked it to do this task but also in it sort of like needs to fulfill these like two competing factors where it needs to be like 100% fair and like unamiguous what
43:54
you want the model to do because you can't sort of like incorrectly fail the model during training because it like successfully uh did what your prompt asked for but your prompt was just not specific enough and so like the model didn't know it actually had to do some extra thing. At the same time, like you
44:05
At the same time, like you need your prompt to be realistic and natural sounding and representative of something like a human would actually ask an AI to do in the real world.
44:12
And it's like very difficult to get this balance right.
44:15
We've heard for like the highest quality coding tasks like the labs are willing to pay well over five figures for a single task.
44:24
And you know, this is obviously already entering into the realm of like how much you would pay a decent engineer for a full week of work.
44:32
And so I think that should sort of just like dispel any myths about this being like easy, you know, mind-numbing work.
44:38
I to all the listeners out there if you're looking for a new job, like if you are really good at creating RL like tasks, you can make seven figures plus annually at this point.
44:46
So maybe maybe consider that as a new job option.
44:52
>> Yeah, that maybe we have some listeners excited.
44:54
Can you maybe actually say like one click lower?
44:58
Do you have personal experience with striking that balance between easy for the AI to do and not impossible for the AI to do?
45:06
Like what sort of intuition could you give to a listener about what that means?
45:12
>> Honestly, it's like it's always changing.
45:14
So, so back in the day, like I actually did sell some environment data to the labs myself. Don't do it anymore.
45:21
uh but it was like much easier to create data even just like uh eight months ago when I was still doing it than it is today.
45:27
I say like today it often looks like you need to identify a specific failure mode that like you are aware of in the model and then you need to create like RL tasks that specifically target that failure mode.
45:39
And really the only way to know if it's like at the right difficulty for the model is like you have the model try solving it 10 times and you see how many times you're successful.
45:47
Um you sort of just like repeat that iteration.
45:49
So, it's like I I don't know if I can really provide any like blanket advice on how to find the right difficulty other than if this is like your first time trying to do this.
46:00
Your first thought is almost certainly too easy, try making it 10 times harder, like try that and then maybe maybe you'll be like in the right level. >> Interesting. Cool stuff. >> Yeah.
46:12
>> Okay guys, we got a whirlwind tour.
46:14
anything you think has uh that I've missed as we've gone through token budgeting, anthropics, profit margins, meta compute, MSL, what uh what have we missed, Crystal, what have we missed? >> I don't know. Nothing I can think of.
46:31
>> Just been too busy watching the World Cup.
46:32
Here we got we got Wednesday, July 15th, right when we're recording this.
46:37
We just got to watch England get knocked out as Argentina stormed from behind for a nice 2-1 victory. That was crazy, dude.
46:43
There's like a bunch of British people in the office right now and they're just like all depressed downstairs. It was crazy. >> No. Joey, how about you?
46:50
What's uh what's what's on your mind as we wrap up here? >> Nothing.
46:54
Um token token spend is up and to the right right now. So, it's good.
46:59
Um you know, I'm I'm excited for hyperscaler earnings starting next week.
47:04
We've got Google, you know, Amazon and and Microsoft the week after.
47:06
It should be pretty good, you know, on the top line.
47:10
>> Okay, let me go around the horn.
47:10
We'll close by getting a vibe check from everybody.
47:13
Joey, what's your vibe like on the market?
47:16
>> On the market or like I like I don't like as long as Lab AR is going up like and at a good pace like it doesn't decelerate.
47:24
I I think like I think things are fine. >> Yeah.
47:28
>> And like right now it's going up.
47:28
Open AAI is catching up like vibes vibes should be good.
47:32
That's not what you're seeing in semis you know the last few days.
47:34
But that's just summer you know >> summer momentum. It doesn't work. So it'll come back.
47:39
People come back to the office from vacations and semis will rip in a year and you know things will be good.
47:46
>> Yeah, we're going to see an acceleration after the summer pop.
47:47
Joey's going to shoot 84 and see Lab ARR go up and be happy.
47:53
Okay, Crystal, how's your vibe?
47:56
>> He's going I have to say I think it's going good. Same thing Joey said.
47:57
I need to leave San Francisco before the AI bubble pops.
48:01
So, very quickly moving away from San Francisco before it's too late.
48:06
>> Crystal, you think it's a bubble?
48:10
Next, we should we should publicly talk about our bet here. >> Yeah. >> Yeah. >> Right now. >> Yeah. We have a the loser.
48:18
So, Max chose the overunder of Anthropic ARR at 400 billion to to end um >> 2027. 2027. >> Yes. 2027.
48:29
And we're at like, you know, we're in probably >> You can't give them the actual number right now, Joey.
48:35
You got to subscribe to the tokconomics model for the actual but you know. >> Okay.
48:41
But just does somebody have who took the over? Who took the under? >> I took the under. >> I took the over.
48:48
>> Jeremy and Joey actually both took the under.
48:50
But we haven't we haven't decided what like the actual bet's going to be for yet. >> No this year. We'll figure it out.
48:57
>> You have to Did you guys see our pot at Rays last week?
48:59
I had >> no >> I pulled out a Canadian 20.
49:02
Rake pulled out 200 Singapore dollars, also some rupees.
49:07
[laughter] We had Dylan pull out some euros.
49:09
Like, we had five different currencies going on the table.
49:11
So, I whatever you guys are, >> I will throw in Canadian currency and uh and take the over with Max because we are >> Let's go.
49:19
>> We're exponential extrapolators here. >> Yes. Yes. >> Exactly. Exactly.
49:24
>> The loser The loser has to write a newsletter post of why they were wrong. >> Okay, that's good. That's good.
49:30
>> No, that's what it was.
49:32
I didn't realize we agreed to this.
49:32
Like I I must have missed that in the in the Slackard.
49:35
>> Are we doing like Frontier Lab fantasy here?
49:38
We need to pick a model for a given week and set up our team.
49:41
>> We'll bet on a we'll bet on Jeremy's spend. >> Jeremy's on Jeremy.
49:46
>> We bet on Jeremy's weekly spend and we'll have like an overunder and >> overunder.
49:50
>> Get everyone involved.
49:50
It's like isn't meta building an internal like Poly Market or Khi?
49:53
Like we'll do that for semi analysis.
49:55
Someone can buy code like Poly Market.
49:57
>> Dude, Meta needs to shut down that effort right now, dude.
49:59
Like what are they doing?
50:01
[laughter] >> Well, we'll have a we'll have a market for betting on Jeremy's tokens. >> There we go. >> I like it, guys.
50:09
There's no way that market could be >> Yeah. >> No way. Totally fair.
50:17
>> Hey, Jeremy, I've got 6,000 rupees riding on you.
50:22
Need you to hammer the data center model dashboard this week, buddy. Okay.
50:27
Well, guys, I appreciate you uh you taking the time.
50:30
Hopefully the listeners enjoyed it.
50:31
Nice uh devolved a little bit at the end.
50:34
Let's uh let's all get back to work.
50:37
Keep tracking those tokens. >> Yeah.