0:00
GDPx taking the world by storm. Something like that. Okay, nice job guys.
Search this video
0:10
Your benchmark matches your what did you say?
0:17
You're pleasantly surprised Our subjective impression of the models.
0:20
Yes, you're pleasantly surprised that the benchmark matches your subjective impression of the model.
0:25
Yeah, because to be clear right, we actually had no idea what the final leaderboard was gonna look like until about two hours before publishing maybe four hours. Right? Because Okay.
0:36
What we did is like we had all the partners difficulty calibrate on four models.
0:42
Like they ran Astra they ran Opus, they ran 3.
0:46
8 Flash and then some Argon, because we were kind of working on the benchmark sort of before Argon came out and then also GLM 5. 3.
0:58
And so we only had results for those four models up until like three four hours before the benchmark actually launched.
1:06
And then Andrew, you know, switched to polyphasic sleep over the past week so he could build all the infra.
1:14
You know, he set up a thing so that Claude would wake him up if there's like something that broke and he needed to like you know provide input.
1:20
But then he like set up all that infra started running on the rollouts You know, these are incredibly long like rollouts right?
1:25
Sort of doing the task plus grading could easily take like twenty-four hours.
1:30
And so we didn't have all the final results until literally you know, three or four hours before we published.
1:35
And we saw this leaderboard and we're like yo, I think you might have quit dude.
1:39
And some of these models like some of these models like we haven't used like or even seen like out in the wild.
1:45
Like Carbon, like we got access to Carbon on like Thursday night or something.
1:49
Like we haven't used it for anything.
1:50
And it's like the first time we're trying it out. It did well.
1:54
It was it was surprising.
1:55
And same with Muse Wonder Full.
1:57
Like Muse Wonder Full was not available in API.
2:01
We just What did you We got access to it and it did great on cyber. Yeah.
2:05
When did you fix the tasks?
2:07
When did those get finalized? Like Thursday night.
2:10
So we launched on Sunday.
2:14
The tasks were finalized like Thursday night because we were doing a lot of sort of back and forth iteration with our partners where like our QA would flag something and be like, Hey take a look at this, and then they'd be like like these two are false positives but actually this one's for real. So we got updated, blah.
2:31
And it was really, really last minute.
2:33
Like, I'm very happy that we got it out in time. It was very hard though.
2:39
I'm also glad to be taking a break now.
2:42
But yeah, it was a crazy process. Yeah.
2:44
I think the work starts now, man. I don't know.
2:49
The work Okay, I'm gonna take a couple of chill days you know.
2:52
And then and Yeah, yeah, yeah.
2:53
Then we can go through the process of yeah I guess kind of reaping the rewards of our effort.
3:02
Maintaining the dashboard as new models get released.
3:05
Did you did you impose this deadline on yourself?
3:08
What was the motivation for that?
3:10
It was mostly self imposed.
3:10
I mean we were we're decently high confidence that you know, a new OpenAI and Anthropic flagship are probably both coming out this week.
3:21
And you obviously cannot launch your benchmark on the same day as, like Fable 5. 5.
3:27
Especially if your benchmark doesn't have it. Yeah, yeah.
3:32
I think we were like I think we were decently confident that like Monday would not be the release date for these models.
3:37
Maybe it's like Tuesday or Thursday or something.
3:40
But if there was like any risk of the model coming out on Monday We were like, fuck it, we're just gonna publish Sunday instead.
3:47
And then there's some fear that like okay, Sunday night, you know, maybe people aren't as online so you're not gonna sort of have the virality component.
3:56
But I think we were very happy with how the launch turned out.
3:59
So yeah, just generally Okay, well generally happy. Take a step back.
4:01
Explain GDPx from scratch for people who maybe haven't seen the article or just hearing about it for the first time or didn't read it as many people who are commenting about it.
4:14
Yeah, I think I'm gonna be going taking like sort of a step all the way back.
4:19
I think it's important to understand that a lot of the really popular benchmarks out there are kind of trash.
4:22
This is largely because they're made by people who don't have expertise in like making evals or RL environments or sort of training data in general.
4:39
And It's like very easy for you to make a task that kind of seems good on a surface level, but it's actually unfair in some way such that it's a poor test of model capabilities.
4:56
A lot of people make their tasks harder by making it unfair rather than actually making the task like difficult.
5:03
And they and it's like very hard to figure this out unless you run like ten rollouts look at the transcripts and see why the model fail and determine like manually like was this actually like a legitimate failure.
5:15
Yeah, can you explain the distinction between difficult and impossible here?
5:19
Like you did this in the article.
5:21
It has to do with both the prompt and the grader.
5:25
Yeah, I think maybe impossible is not quite the right word.
5:28
It's more like difficult versus unfair.
5:30
I think one classic example of a task would be unfair is if you have like a coding problem that asks the model to fix some bug maybe in some open source repository.
5:41
And then you could have sort of an expert human software engineer look at it and they would say, like, there's sort of two ways to fix this bug.
5:47
You can either use like module A or module B.
5:49
And I, using my human expertise recognize that both are kind of equally fair equally valid.
5:55
Yeah, you should be able to pick whichever one you want.
5:57
But when we're making these benchmarks you also need to include a verifier for each task so you can check if the model succeeded or not.
6:02
And it's totally possible that some of them might make a verifier that's sort of scoped to just the module A solution.
6:10
And then when sort of AI comes and tries the problem maybe it will sort of rightfully pick module B instead of module A some of the time.
6:17
And then the verifier would mark that as a failure and sort of you know, the AI would Perform worse on the task.
6:24
I think this is an example of the task being unfair because it's like the model did technically solve the problem.
6:31
It's just that your grader is not faithful to the prompt.
6:35
And it turns out there are actually lots like most popular benchmarks out there like Terminal Bench 4 or SWE-bench Pro or DeepSWE or stuff like that, they actually have tons of issues of this type that people have kind of funny enough started really noticing over the past like month or so.
6:56
There have been a couple kind of big blog posts on this.
6:58
Because it actually just it's actually a lot of data companies who were sort of just have agents look at all the transcripts look at all their prompts, sort of run the tasks for their own QA process and they see like, actually sixty percent of the Terminal Bench 4 tasks are broken or whatever.
7:13
The funny thing is like I feel like everybody knew it.
7:16
Like all the labs definitely know it.
7:17
Anyone who looks at data, like definitely knew that the benchmarks are broken.
7:20
It's just like no one really like spends the time to like write about it.
7:23
Because again, like to like most people in the industry it's like not it's not new information. It's like unsurprising. Yeah.
7:27
So, public benchmarks are bad.
7:31
That's sort of one thing to know.
7:33
And then we at SemiAnalysis, like we knew we wanted to make a benchmark ourselves because it's kind of the only way you can consistently say like interesting things about new model releases.
7:43
Because the team is not gonna sort of spend meaningful time and effort trying out I don't know, the new MiniMax model when it comes out to see if it's actually good or not.
7:54
No disrespect to MiniMax but like, you know, we're just not gonna do that.
7:58
And so you really want an eval.
8:00
And we're not gonna do like no one fucking tested you know, GPT 6. 1 Terra either right?
8:04
But if you have sort of a really high quality eval that you can normally do models on, it can go a long way in sort of tracking the performance there.
8:11
But at the same time, like we also recognized that a lot of public benchmarks are bad sort of Not because of lack of effort on people's parts.
8:26
It is actually also just really hard to make like a task that is fair but also sufficiently difficult and also interesting and reflective of like real work.
8:35
And it would have been sort of very naive of us to think that like SemiAnalysis is gonna be able to come up with like 200 really high-quality tasks ourselves.
8:43
And so the strategy we took is we partnered with 14 different sort of specialized data companies who use like a lot of their expertise to actually supply the task for each domain.
8:58
But we were responsible for sort of actually QAing all of them.
9:03
We actually we like dropped more data providers than we actually included in the final benchmark because a lot of them just failed QA.
9:10
And also we are sort of very heavily involved in with like task iteration for the ones who pass QA as well as we kind of mentioned at the start of the episode.
9:18
That was sort of like I guess the philosophy slash methodology for how we actually made the benchmark.
9:25
In terms of what the benchmark covers this is obviously sort of another kind of big question we need to figure out because you can I mean you can you can kind of pick any domain right?
9:37
AI is affecting everything these days.
9:38
But one of the things that we sort of pretty strongly believe on the Tokenomics team is that the first trillion dollars of AI revenue are going to be highly concentrated within a few domains that can actually absorb sort of increasingly smarter frontier intelligence and earn high ROI.
10:01
I think a lot of people recently have seen sort of these token budgeting trends or people talking about how like, you actually don't need frontier intelligence for the majority of tasks and they view that as bearish for the AI trade.
10:16
We sort of totally agree with the fact that you know most tasks a lot of tasks will not need frontier intelligence but we think there are gonna be a lot of domains like chip design or biology or cybersecurity where we actually are just gonna need sort of the increasingly smarter frontier model and you will continue to get high ROI from Fable 6 table seven, table eight, whatever.
10:39
And so we decided to pick domains for our benchmark GDPx, which sort of match these.
10:46
Domains that can absorb frontier intelligence.
10:50
And that was kind of our guiding philosophy there. Sweet. Okay.
10:55
Maybe we can talk through the specific domains in a little bit more detail.
10:59
For those who don't know there's a website up, gdpx. semianalysis. com.
11:05
I'll share it on screen here for a sec.
11:08
And as we scroll through the domains recursive self-improvement or RSI chip design, cybersecurity, biology finance, legal, quant trading hardware design, material science alignment, and software engineering.
11:19
I think this is not really a breakdown that a lot of people have on their tip of their tongue when they're talking about like the domains AI is coming.
11:27
So even just introducing each of these one by one I think would be interesting.
11:31
But maybe you can start by saying how did you arrive at these categories broadly?
11:37
And then how did you find the specific data vendors that you were going with in the given categories that you're focusing on? Yeah, yeah.
11:46
So arriving at the domains was a lot of just like sort of brainstorming ourselves trying to pick the domains that are verifiable and also the ones that seem like they would have sort of the capacity to absorb a lot of intelligence.
12:03
And after we sort of had our initial list we kind of talked to a bunch of people at the labs people sort of working on similar problems and then we We kind of did this process for maybe a few weeks until we finally arrived at our final set of domains.
12:18
In terms of how we pick the partners we're sort of fortunately decently well connected in the data space.
12:29
And we actually just happen to know sort of a lot of the RL environment companies and what each of their specialties are.
12:34
So we reached out to sort of a bunch of people who work in the various domains.
12:41
We got sample tasks from them we QA'd them, we sort of dropped the ones that are bad.
12:46
In the cases where we had like more than one good person who wanted to be exclusive with us in a particular domain we would tie break based on things like taste and how interesting we thought the problems were and representative of the particular domain.
12:59
And that was yeah that was basically the whole process there.
13:05
Eventually people actually started like hearing about GDPx like on their own and started reaching out to us.
13:10
Be included in the evaluation.
13:12
It kind of spread around like the data space. Yeah, yeah.
13:14
Some of them were more opportunistic like alignment is a good example where it's not something you'd probably immediately reach for if you're trying to make a benchmark that measures agent capabilities.
13:28
But I really do think it is like one of the most important things that you need to evaluate future models on.
13:31
And so it sort of deserves its place in any comprehensive benchmark.
13:38
At the same time, Fleet is sort of one of the biggest like most well respected data companies today.
13:43
I have a few friends who work there.
13:46
And so we're when we were sort of brainstorming domains like that we could potentially include Fleet for alignment and it's just being one like made a ton of sense. Yeah. Yeah, makes sense.
13:57
Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like first of all, these are clearly domains where people using frontier AI today are burning a ton of tokens. Absolutely. Sink. Okay.
14:12
The domains are really interesting though because like it's it varies so much.
14:14
Like some of these are like very established data companies who've been selling data for like a few years now.
14:21
But then some of them like material science are just like one PhD who's just like really, really good at material science.
14:27
And they just like delivered like these insane tasks that are difficult enough for like even Astra.
14:31
And like I found that like super cool.
14:36
Because they just use like papers that like Yeah, right.
14:38
They wrote or like papers that like people that they know like wrote. So shout out for those.
14:42
Like those are pretty cool evals.
14:45
Yeah, my literally my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much it was just such a it was so clear by reading through the details of the task with the person who had written it.
14:55
Felt to me like a domain expert.
14:58
They certainly knew more than me.
15:00
And maybe you can talk a little bit about the reaction from the guys that you worked with since the release of the article and how it seems like a lot of people were Surprised by the results.
15:10
Like I know Bryan was specifically saying like he's shocked that some of the tasks were solved by the frontier model. Yeah. Yeah.
15:18
I think Bryan had like a brief moment where like you know, initially like he didn't actually expect like Astra to be so good at his tasks when we first like asked him for stuff.
15:28
He's like, actually, yeah, some of these workflows that I initially thought Astra was not good on, like once he sat down and made tasks he realized that actually no, like I was just prompting it wrong or something.
15:39
Or I didn't provide enough context and it just ends up solving.
15:42
So you have to make like another set that was like actually difficult and frontier level.
15:47
'cause I think a lot of people don't realize how good the models are.
15:49
And he even said this, like he said like these material scientists like don't know how good the models are.
15:55
And like they think that the models can do certain things but really they can't.
15:57
Well, you should talk about like the experience you had with software engineering at the start of the year and then how that Yeah. Relates to Bryan.
16:05
Yeah, I think I went through the similar thing like essentially a year ago at this point.
16:12
Like, you know, after I left My job to start like an RL company with Max here.
16:21
Like at first, like, you know I was still of the opinion that like humans plus AI is just like a lot better.
16:27
And you know, AI can't really take software engineering job like we'll still be coding by hand like the but like you know, AI will automate the easy things but eventually it just became like you know, Sonnet 3.
16:34
7 was really really good.
16:36
Sonnet 4 came out Sonnet 4. 5, and then Opus 4. 6 and that like Opus 4.
16:37
6 was like a moment 'cause that like weird. Opus 4. 5 first one. Opus 4.
16:41
5 was really good but Opus 4.
16:44
6 was also really And like those two models just like Yeah like I think, you know, give it a few more iterations like there'll be no need for me to look at a lot of this code.
16:54
And it was like a very clear trend and I had to like reconcile that myself because I really like coding at the time.
17:01
And that was like my thing.
17:02
Like I was just like good at coding and systems but yeah, I just had to like figure out like actually, you know, I have to take the higher level things now.
17:09
Like this is no longer a job that like I'll always be doing like writing code by hand.
17:14
Though I still think like, you know human plus AI is still better than AI alone just because like this like little this piece of entropy that you can provide that the models don't have themselves.
17:24
And so, you know, it's still valuable. Yeah.
17:27
But f but funny enough Do you feel like that the reason why I wanted Andrew to talk about that is I think Bryan had a very similar moment for material science when he made the task for a benchmark.
17:38
Because he was like Holy shit, you know, if Any human material scientists could do what like Astra just did.
17:48
They're probably, you know, a top like they're definitely better than the median material scientist already.
17:51
And like and like I just watched Astra do this.
17:56
And so I think we're probably going to sort of have these similar moments in lots of other in domains like chip design and material science and sort of the other GDPx domains in the imminent future.
18:12
And that's something that you also hopefully be able to see sort of in the actual benchmark results as we're able to evaluate you know, Fable 5. 5, Astra 6. 1 6. 2, or whatever.
18:21
And I think we got like a really good tweet.
18:24
Someone asked about whether like you know GDP Val had human baselines for how long tasks take to like put things in context.
18:32
And this was really funny because like I responded to this and I was like actually, like it doesn't even make sense to have human baselines anymore.
18:37
Like in the material science tasks like Astra solved a subtask of a task better than like a published state-of-the-art reference.
18:46
Like when the models are like essentially past expert level how do you even put a human baseline to that?
18:53
And some of these chip design tasks are just so long horizon and like would take teams at companies to even like do and like those like there's no human baseline there.
19:03
Like, how do you even put a number on it? It's a lot of money.
19:07
Whatever the hourly equivalent is in human work hours is a lot of money.
19:11
And I just threw up Yeah.
19:13
On screen the cost per task for the material science work where the biggest ones like was at forty or fifty dollars per task and many of the models are solving these tasks in some of them in ten dollars, which is I think a pretty unbelievable statement.
19:33
Maybe you can talk about Like reading some of the rollouts in more detail?
19:38
Like what was what were some of the pieces of model behavior that surprised you the most?
19:45
Yeah, I mean the models find very creative ways to solve the tasks.
19:49
And like this is again like going back to the whole unfair verifier thing.
19:53
People don't expect the way that models solve tasks because models are not human.
19:57
They just solve things in very weird ways like you know, maybe use module B instead of module A for example.
20:02
But sometimes just like have completely different taste on like how things are done.
20:07
And sometimes it's correct and sometimes it's not.
20:11
And that's like that's why we have to like go through this transcript and It's like hard to even go through them sometimes because you know, we're not human experts but luckily, you can benefit a lot from chatting with AI about certain rollouts to like figure out whether stuff is like correct or not.
20:30
The model's actually so smart like on their own that with a with human probing you can figure out whether something is like fair or not fair.
20:39
It's a very interesting experience for sure.
20:44
Maybe we can talk about the results.
20:46
I'm getting I'm realizing that I haven't even shared the Yeah.
20:50
Like flagship leaderboard screenshot that's at the top of the article yet.
20:53
Maybe you guys can talk about like what you learned about the models themselves.
20:58
I think particularly people we've been going back and forth and trying to explain the difference between like the flagship leaderboard across all domains and then compare it to the next one where You've got the leaderboard where you filter out biology cybersecurity, and alignment where safety classifiers are causing models that have better performance, you know in the weights somewhere to not actually demonstrate it on the tasks in the benchmark. Yeah, yeah.
21:28
I think that's a really good point worth emphasizing because we kind of went back and forth between which of those screenshots was the one that should have been the headline result.
21:41
The one where you include all the domains versus the one where you exclude cyber biology and alignment because those were the three domains where sometimes safety classifiers kicked in.
21:54
And therefore sort of sandbagged the score for some models.
21:57
Like if you look at Astra on cyber for example, the score is like zero percent because it just refused to do any of the cyber tasks.
22:03
And ultimately we decided that the one including all the domains is better.
22:10
But I think it actually is sort of fair to penalize overly strict safety classifiers to some degree.
22:20
I mean I know you've experienced this yourself Jordan, so just Doing your day-to-day work at SemiAnalysis and then you're hit with a cyber safety classifier from like Opus or Astra and like, well, fuck, I guess I gotta use like GLM 5. 3 now or whatever.
22:32
No, Jordan, Jordan was Jordan had a very interesting insight.
22:35
He actually figured out that Muse Wonder is like a top-tier cyber model independently. And you were right.
22:41
Like you were totally right.
22:41
Like Muse Wonder and Muse Wonder Full were just top tier for like what you can get out of the box for cyber. Yeah, yeah, yeah.
22:50
That which was again a very surprising result for us because I guess Jordan already knew, but we know we didn't have the Muse Wonder Full number until two hours before launch.
23:00
So that was that was pretty cool.
23:01
And we were like, Yeah, wow, Muse Wonder Full just crushed it.
23:04
Yeah, you guys you guys gave away all my alpha there.
23:06
Look, using the models yourself for something that you think you're pushing the frontier on absolutely gives you insight into which models are good and which models are bad, but you have to actually have that work to do.
23:17
And I had that work to do in one of the 11 domains we're talking about here.
23:22
So I could yeah, I could figure Yeah, yeah. That out.
23:25
I and I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as it is relevant to you.
23:37
as it is relevant to you. People with Daybreak access or with CVP access from Anthropic, which we have sometimes Well there's like three or four tiers and then we only had like you know, the bottom or maybe like the second to like worst tier and actually
23:56
that tier I think is more or less useless because you still get flagged doing basically everything and it's just I think they both have like very confusing sort of access programs for them all today which is hopefully something they can like you know, get better at. The The Yeah, yeah.
24:11
Crazy thing is like, you if Muse Wonder Full does like that well on cyber like I wonder like what red access on, like, Astra or But no one can use it. Yeah.
24:21
Unless you work at like Palo Alto Networks We'll get there. We'll figure it out. I guess. We'll get there. Yeah.
24:26
But okay, maybe the better question about this is just like maybe you guys can emphasize again the fact that this benchmark is meant to measure the performance of the models at the frontiers rather than on day-to-day tasks. Yeah, yeah, for sure.
24:41
Because again our core thesis here is that day-to-day tasks just don't actually matter. Right. Okay.
24:48
They're more or less solved.
24:48
Yeah like I think I think most of them the majority of them are like more or less solved.
24:54
I think heading into 27 you're not gonna need frontier models for the most of these.
25:03
And we like not only do we think they're sort of Core tasks to use to measure frontier model capabilities.
25:12
We also think it's probably gonna sort of not be the majority of revenue for OpenAI and Anthropic and the other frontier model companies.
25:21
And sort of because the thing that we're most interested in tracking with this benchmark is sort of A, where frontier model companies sorry, where frontier model capabilities are going.
25:33
And then B, can you sort of like do any like TAM or actually revenue analysis based off of that.
25:40
We sort of very explicitly focus on just the frontier domains.
25:46
Yeah, and like when we say like day-to-day tasks like there actually doesn't even exclude like some of the domains people want to use like these models for.
25:51
Like it was it was surprising to see how bad the models are in like finance and legal for how much people talk about Like they will automate finance and legal.
26:02
Cause like when we looked at some of the tasks that we got from like living context and chakra, like we went through those rollouts ourselves because we were confused.
26:09
I actually thought the models were better at these Yeah and Andrew's like a tech bro supremacist and he was like, dude, the model's definitely saturated finance already.
26:17
That was his prior coming into this.
26:19
Yeah, and we actually had like some of our finance friends look at some of these tasks.
26:24
And like when we were like choosing partners for these we you know, we had different rollouts and tasks and like some of them were like yeah, these tasks are like pretty bad.
26:33
Yeah, like there are some tasks from some potential finance partners where we were like, whoa, this is like not realistic of sort of actual finance work.
26:41
Like, and the only reason why it's sort of sufficiently difficult for you know Opus and Astra is because it's like lacks realism and is also unfair in these ways.
26:51
And then after you correct sort of unfairness like look, Opus like saturated.
26:56
So that would make you think that these tasks are solved.
26:58
But then we got like the really high quality tasks. Yeah, yeah, yeah.
27:01
And then like we had like our finance friends look at them and they like, yeah, these are this is totally realistic of like a real thing I would have done in like banking and PE.
27:10
And the models just like failed us.
27:12
Yeah, and actually I want to talk more about the finance task in particular because I think it's probably the most understandable kind of domain to the average listener.
27:21
And I think this sort of be useful.
27:25
And giving people a sense of like why GDPx is such a high quality benchmark.
27:31
Sort of how people have historically made finance tasks in the past if you look at something like GDPval from OpenAI is they would go to say a company like Mercor which has this vast like expert network of finance bros and then be like, you finance bro sit down for two hours and try to like come up with the task.
27:53
That's like representative of something that you would actually do in your job.
27:56
And then also give me sort of the golden solution for that task.
28:01
And then you know that's gonna be the task that I'm gonna give you though.
28:03
And obviously, you sort of have a ton of sort of biases when you're telling someone like, Hey, you know, make a task for my for my benchmark and just please try your best to make it realistic man.
28:13
What our finance partner did is they took a real private equity transaction.
28:23
Where like some private actual firm bought a like family-owned snack company.
28:27
I think they paid like a little over 800 million for like 62% of the company.
28:32
They again they sort of had this actual transaction like all 4,000 files sort of corresponding to it separated into the deal rooms for the private equity firm that was involved the investment bank that was involved the accounting firm that was involved and then also sort of the snack company's own VDR because you know they were selling themselves.
28:52
And it was organized or disorganized exactly how they would Yeah, it was like it was like the real the real like files for this real deal that Yeah, I was looking at this.
29:01
And anybody who has worked in a VDR like a data room for any M&A transaction has seen exactly this and it's realistic. Yeah. Yeah.
29:10
So the so it had the they had the real files.
29:13
And then obviously they sort of like de-anonymized it.
29:17
But along with the files, it actually has sort of all of the real tasks that you know, analysts and associates and other people like working on this deal had to do.
29:25
And they essentially converted this like you know, actual deal room into an environment they can evaluate the models in.
29:34
And then they like found the emails.
29:38
That you know some accounting partner told their associate to do as part of this task and then they had the model do it.
29:45
I think this is this is obviously sort of just like a categorically different thing in terms of realism and interestingness and quality versus like just taking some random, you know banker off the street and being like you come up with the task.
29:59
Okay, one more thing about this Max, is the life cycle.
30:01
I thought this was fascinating which is like they could effectively replay what the data room looked like when Exactly. Yep.
30:08
That task was assigned, right?
30:10
So like you're gonna add in these thousand files into the data room over time, but not every task is assigned at that same time in the data room.
30:17
I thought that was like yeah, obviously super fascinating that this is the depth at which people can go to Yeah.
30:25
To try to make this task realistic.
30:27
And some But of the failure modes are like so like esoteric when like I looked at them.
30:32
But then like I had to ask like my finance friends and they told me like yeah that you would totally be expected to just have done this.
30:37
It'd be like some email from somebody who was like No, actually adjust these numbers to like whatever Q4 or something.
30:45
It's just one email where it mentions this.
30:47
And if the model didn't find that email it would fail because you know the it had to it had to find that email.
30:52
Yeah, like actually evaluate all the context relevant to the task.
30:55
And the models didn't look hard enough sometimes.
30:57
And so just miss it and then like have the wrong numbers go out.
31:00
And if the human associate failed they would have gotten yelled at.
31:02
So it's Yeah, apparently that's actually that's exactly how it would have worked.
31:05
Like if you were if you were an analyst and you did not if you didn't see that one email.
31:09
If you were not on top of it you would you would have you would have gotten yelled at for sure.
31:13
They would have said, Yeah, this is unacceptable behavior in a real banking environment.
31:17
I know some people on the Tokenomics team who I think are possibly going to be listening to this podcast having some flashbacks from thousand-yard stares. They're gonna say PTSD. Yeah.
31:26
Can you talk about the grading in this case?
31:27
Cause I think a lot of people have been confused about the partial credit versus binary grading approach that you have to apply Yeah.
31:36
To some of these things that don't necessarily have a binary outcome. Yeah, yeah.
31:40
So in the case so I'll say our finance tasks use binary grading.
31:46
So the model either gets a zero or one.
31:48
But there's sort of a continuous sort of scoring component there.
31:53
Because you might have a task where the final deliverable is like some Excel workbook.
31:58
Maybe it's like a quality of earnings book that like the accountant had to make for this private equity deal.
32:07
And obviously, like that workbook can kind of be decomposed into a bunch of different parts, you know, different cells and the model has to find the right number for.
32:15
And you can kind of assign partial grading for each of these atomic criteria.
32:22
But then we still use kind of binary grading at the end because if you were a real accounting associate and then you gave your boss a workbook where like half the numbers are right and half of them are wrong.
32:35
They're not gonna be like, man like great job, little Jimmy, like you got half of it right.
32:40
Like I'll take from here.
32:40
They're gonna be like dude, what the fuck, man?
32:42
Like I'm firing your ass right now.
32:46
So I think in those cases, like it makes sense to do binary grading even though there is like a partial score component.
32:52
So yeah, maybe Jordan, I'm gonna show here that like we have zero percent when you use binary on the finance basically except for Carbon who kind of got like one task right.
33:00
But then you switch to partial and you sort of get much more signal on how the models actually did.
33:06
That being said, okay, actually I'll say one a lot of people on Twitter were kind of shitting on us using binary grading because there are other benchmarks that kind of use that to hide the fact that their tasks are actually quite easy.
33:19
I won't name any particular names here but there are some popular benchmarks where you look at binary grading Yeah.
33:25
And it's like, 25% on Opus.
33:28
Like this is probably pretty hard.
33:29
But then you switch to partial grading it's like 97% Opus.
33:31
And it's like okay, like That last 3% is probably just like some esoteric requirement that there's no reason you should have expected them.
33:40
Yeah, or your task is just like fucking unfair and broken in some way.
33:42
And yeah, so like I agree.
33:42
If you look at our finance task and it was like 95% on Opus or partial grading they're probably a bad task.
33:48
But you saw it was like fifty-something percent. Same with our legal.
33:52
The legal is all zeros but then the best model, even when you switch to partial grading I think is like it's in the thirties or forties.
33:59
At the same time Most of our domains actually use continuous grading because it makes sense for those.
34:05
You know, you have a chip design task where maybe the model is sort of asked to do some like PPA optimization.
34:13
And then you just have like zero as some naive solution and then you have one as sort of some like max theoretical achievable solution and you kind of just give the model partial credit depending on how much it can optimize like your chip design.
34:26
We sort of picked the grading strategy that like makes the most sense given the particular domain.
34:33
We did not do, you know binary grading for everything so we could kind of artificially lower the score.
34:41
Yeah, and I think this is maybe goes back to the comment about frontier intelligence versus day-to-day tasks.
34:47
I mean, I don't know what frontier in finance necessarily represents the same way in science, where it's pretty clear like you're pursuing a new discovery Yeah. Or something like that.
34:57
But finance like the frontier is being accurate and Yeah, I think I think one of the things to really note for finance is that maybe sort of at first glance you think it's like different from you know, chip design or material science.
35:14
But we actually think it is going to be one of the domains that sort of spends very heavily on tokens in 2027.
35:20
It is actually already one of the main domains that people spend tokens on in 2026.
35:23
Largely because like the margin structures for the businesses sort of supports it.
35:31
Like they can actually just afford to spend a ton money on tokens.
35:34
But then also there are and this is actually this is even more true for legal but there are some sort of like PvP components there where if we if we take legal for example, if the AI sort of could theoretically do a better job like winning this lawsuit or like redlining some contract like that actually does have implications especially if like maybe your like opponent in this scenario is not using the superintelligent AI to do these things.
36:04
So I actually think finance and legal like are domains that will be able to absorb a lot of frontier intelligence.
36:12
Whereas maybe things like real estate or like government or something might not be. Makes sense.
36:21
And just to like if you could expand on that a little bit more, because I think I know the backdrop you're specifically talking about the reason for focusing on these domains specifically rather than like the choice to do that rather than just taking a cross-section of all knowledge work in the economy and saying we're gonna have tasks for every single thing that people do across the economy right? Yeah, exactly.
36:47
So maybe you can I think the thing that the listeners are probably listen thinking about right now is just the concept of AI-driven job loss.
36:57
If people are using frontier intelligence and they're gaining these productivity improvements or they're making more money or they're you know winning more things, maybe you can talk about like how does running through a benchmark like this inform your point of view on whether AI is going to Yeah, cause job loss in the next year two years, even five year time horizon.
37:21
I actually think they're like orthogonal in some sense because we don't we explicitly choose not to cover a lot of sort of the white collar knowledge work domains that people are probably most afraid of drop loss occurring, like I don't know, customer service or something like that.
37:44
In terms of whether or not we will have like massive job loss from AI it's really hard to say.
37:49
I think a lot of the doomers have probably been very pleasantly surprised by sort of how AI plus human like co-working has evolved over the past year where it's like every any software engineer that sort of like has their eyes on the ball will probably tell you that Opus when given sort of like a well specified problem is just like a better software engineer than me, period.
38:18
But at the same time like good software engineers are more valuable today than they've ever been.
38:25
Now, is it possible Yeah okay.
38:27
We're sort of in this like Goldilocks zone where the humans can like a lot of humans can still provide kind of that like taste and supervision and judgment that makes it so that like them plus the model is like way way more effective than just the model.
38:46
But then like maybe two years from now you actually only need like one dude at each company who's like supervising all the software engineering and like basically everyone else got fired. Maybe. I don't know.
38:54
I think it's possible that's the world we end up in.
38:56
But it's also possible it's not.
38:59
I think we just have to You really gotta pay close attention to the model capabilities to see.
39:04
I'll say it's definitely raised the bar for the kind of person you're looking for to like work like on stuff.
39:14
Like w when you when you're looking at like software engineering for example, like I feel bad for the interns. Yes. Essentially.
39:19
Like you have to be really good to like provide value higher than just pure like Opus at this point as a software engineer.
39:26
I think a lot of people viewed interns as like I'm gonna g These are the people who I give my well specified well defined problems that I don't want to do and they're gonna go do them for me.
39:37
But obviously now you just tell Claude to do that.
39:39
And then sort of all the additional overhead of managing an intern oftentimes becomes sort of just like unbearable in those cases.
39:47
I know one of the big quant trading firms had a post around this.
39:50
It was like, yeah, really?
39:50
Yeah they said their interns like finished their projects like in the first like few weeks this year compared to previous years.
39:59
Hey, at least interns know how to use AI then. That's good.
40:03
I think the typical response from people is that you need to get more ambitious. It makes sense.
40:07
I mean, going back to cybersecurity or I've been doing some work recently.
40:12
We saw this recently, just with the increase in CVEs.
40:17
I posted this on October first. It's been a month.
40:19
I don't have enough data to prove there's a lot more than just this at this point. Yeah.
40:26
Dude, is the whole internet just like broken it's kind of promising. Now?
40:28
Dude, what's gonna happen?
40:31
What are you gonna do, Jordan?
40:33
Well, I think it's a the yeah I can go I mean into that in a lot more detail.
40:38
Let's save that for another time.
40:40
But let's use it as a way to make the point that there's I think there's a lot more demand for cybersecurity experience right now even when using the models. And Yeah. For sure.
40:52
I think that's one of the first domains as a product of software engineering that's getting knocked down, but If you're pushing the frontiers of material science for example, you're probably not automating away the jobs of the material scientists.
41:05
You're probably starting to discover a whole bunch of new materials that need to go through the whole discovery phase and a lot more people.
41:12
Well no, I think it is very possible that you have automated the job of your like PhD candidate assistant if you're like a cracked material scientist.
41:20
I think this is actually exactly what we've heard from the partners. Yeah. Bryan, even.
41:25
Yeah, if they if they have no ambition to go beyond what they are doing today or last year, then yeah, there's nothing useful for them to do.
41:33
But if they have ambition to accomplish more with the tools that are at their Yeah. Yeah, yeah. If the disposal.
41:38
If the person who would have been forced to just be you know, kind of some PhD candidate grunt worker two years ago can now kind of leverage their infinite PhD grunt workers on tap to elevate themselves and you know, just do way more interesting material science that's great.
41:54
We'll just have to see sort of I'm I think I'm generally very optimistic kind of human agency and people being able to kind of take their life in their own hands and like you know, do whatever makes them happy.
42:09
And so I am just kind of bullish people kind of using these tools to be just up leveling themselves essentially.
42:17
We'll have to see what actually happens. It's interesting.
42:19
I actually completely disagree with that. Andrew's a hater.
42:22
And that I think most people Well I mean when you look at it like most people in America they just want to go in put their hours in and go home.
42:30
They like don't care that much about their careers and like And like me maybe actually like this is what forces them to care.
42:35
'Cause like otherwise like you won't be able to provide value, but you know, we'll see. Interesting. Okay.
42:43
Let me go back and get some quick reactions to results that we kind of glossed over.
42:48
So Andrew you love Opus 5. 5.
42:52
Were you super happy to see it come out on top on a lot of these leaderboard charts or feeling justified? Yeah.
42:59
What where'd you get that insight?
43:02
I mean obviously like you know I'm glad that my subjective experience was validated.
43:08
Though, you know some domains that I'm not particularly involved in like it was very clear that Astra was just like ridiculous ridiculously strong on.
43:18
Something's like material science but also, you know, like some of the engineering stuff.
43:24
It was it was very clear that like Astra was like a step ahead there.
43:27
And maybe that's just 'cause we don't have Fable 5.
43:27
5 yet but you know, we'll see.
43:31
But it's good to see that like different labs have different strengths.
43:36
Because you know they spike in different things so you can leverage whatever is best for what you're actually doing. Yeah, makes sense.
43:44
How about Gemini 4 Carbon? Quick take there.
43:48
We saw Sundar quote tweeted everything there.
43:52
You guys are excited about that Like Yeah, that was sick. Shout out Sundar.
43:59
I mean, we're excited about it too.
43:59
Like, hey more frontier models is better for all of us.
44:03
We definitely want, you know better models from everybody.
44:06
It only benefits the consumer as there are more competition.
44:10
Yeah, I do I do however think that the right sort of apples to apples comparison for where the frontier labs currently sit is if you exclude cyber and bio and alignment and look at that GDPx score.
44:27
Maybe even switch to partial scoring too because that kind of gives you some more signal on legal and finance where no one's smart enough to do binary grading.
44:35
Or at least get reward for binary grading.
44:37
And if you do that sort of ranking you see that Carbon is still kind of behind Astra and Opus etc.
44:45
But that being said they it is certainly sort of An inflection in the trend for where Google's model releases were going.
44:56
Because I think at one point it kind of felt that they were gonna fall farther and farther behind even like Meta in particular potentially even xAI, and potentially even some of the Chinese open source guys.
45:08
But they've like kind of really reestablished themselves as the third place player with Carbon.
45:12
And that was definitely impressive.
45:15
How much of this is a timing thing then?
45:17
Like sorry to interrupt but Fable 5.
45:19
5 you guys are teasing that might come out soon.
45:23
Obviously Opus, Sonnet are on the list there a lot. Yeah, yeah.
45:27
So I think I think the gap between Anthropic slash OpenAI versus Google today is likely still very big, because like we're gonna have new flagship model releases of them quite soon.
45:40
But like Muse Wonder Full came out pretty recently right?
45:44
And so I think you're sort of evaluating Google's position versus Meta and xAI.
45:50
I think there's a pretty strong argument to be made that actually they're like the top of that kind of tier two cohort.
45:58
But I definitely still believe that there's like a pretty meaningful gap between OpenAI slash Anthropic versus everyone else.
46:05
And you know, there's like the thing like you know, OpenAI and Anthropic are obviously not releasing their best models.
46:10
So it's just a comparison between the public models.
46:14
Well like Gemini and like Meta like they are releasing like the best stuff they have so far. Yeah.
46:18
Anthropic could just be on like Mythos four at this point. Who knows? Yeah, true.
46:24
We're gonna get like Fable plus one.
46:27
Cause I think Anthropic is still kind of rolling with Fable One level models essentially. Or like Mythos one.
46:34
I guess it's called Fable 5.
46:35
Internally they're probably like at least two generations above that.
46:41
And then we're finally gonna get you know, one generation above that hopefully this week.
46:44
So I think the gap between sort of the actual best model each lab has, including internal versus Well actually that gap is just sort of very wide across OpenAI Anthropic, Google, Meta, SpaceX. Yeah.
46:59
Well, maybe the biggest takeaway for me there is just how relentless the pace of model releases is.
47:07
And I mean the trend is for it to get faster and faster where they improve post training and improve the data mixes and just keep going. RSI, bro. It's RSI. Yeah.
47:17
Can you talk about next steps?
47:21
Like where do you go from here? You think it's possible?
47:23
Actually one thing we didn't talk about at all is the concept of releasing one task and one rollout for the category for each of the categories or well for each of the data vendors who multiple from some of the categories.
47:38
Can you talk about that approach that you guys took there of the mix of kind of partially public, partially private and how other benchmarks approach it by comparison? Yeah.
47:49
So I would say the trade-off that you have to make for any benchmark is the more public you make it, you know, obviously the more experts are able to QA everything themselves and kind of verify that your stuff is actually good.
48:03
At the same time, if you make it public it just becomes like way too easy to hill climb on the benchmark.
48:10
And oftentimes like your benchmark no longer becomes a measure of like model capability and then more becomes a measure of like which lab bought more data that looks just like the tasks in my benchmark.
48:25
And again I'm not gonna name any particular names but I think there are a lot of public benchmarks that have already sort of like fit into that category.
48:31
It's really just a measure of like okay, you know, which lab spent the most effort to buy the most data of this shape.
48:38
And priority number one for us is that we don't want people to be able to hill climb our benchmark.
48:45
We sort of want our benchmark to be an accurate measure of sort of real model capabilities.
48:49
But at the same time we obviously know that establishing credibility is important.
48:55
And so releasing like the rollouts for one task per partner we thought was a good trade-off there where kind of people can get some visibility into each partner and how confidence their tasks are good.
49:06
At the same time, sort of us going pretty in-depth in the actual methodology for like each partner and explaining how like You know, our finance guys actually use this like real deal room.
49:18
Our cyber guys like found their own zero-days in front of them the task.
49:21
I think sort of binding all those details there is the other way that we can sort of instill confidence in the quality of the benchmark.
49:29
Also just like giving details in our QA process where we say like, yo, you know, we ran our own agentic QA then if you take Terminal Bench 4 literally flags like 50% of the tasks is broken but then we ran it on every partner and we made sure that like if you couldn't pass this bar, you just weren't included in the benchmark.
49:44
I think stuff like that is what allows us to sort of give people confidence in the quality of the bench without making all the tasks public.
49:52
As for what's next, we well we really hope we can get sort of version one on some major new model release cards when they come out because we do think it's sort of like the best benchmark for measuring frontier capabilities in the world today.
50:07
At the same time, we're definitely like just getting started.
50:11
And we plan to do like future versions as models become increasingly capable.
50:17
They saturate the tasks and GDPx v1.
50:20
There are sort of more domains that get unlocked that can absorb lots of like frontier tokens and like be super valuable earn higher ROI.
50:29
And so we definitely want to release like future versions with more, you know, data partners always using sort of the highest quality data partners at any given point in time always picking the most important domains at any given point in time.
50:43
Hopefully we can do things like have more partners and have more tasks per domain and also run all the models at every reasoning level and stuff like that now that we have like you know more clout and more traction with the benchmark. Awesome man. Okay.
50:59
I think you answered the final question about where we go from here.
51:02
Andrew, any like closing thoughts from you about the future next steps, initial reaction from this stuff? Pretty fun week, huh?
51:09
Yeah, I mean definitely agree with Max.
51:10
Like I do want this to be like the benchmark everyone thinks about when they think about like what model has which capabilities and where they stand.
51:18
And this is only the first iteration.
51:22
Like we're gonna we're gonna keep working on it.
51:24
We're gonna get hopefully more data partners.
51:26
And if anyone listening to this is interested in evals and you know, maintaining this, please reach out.
51:32
We are hiring on this team, of course.
51:35
Just email gdpx@semianalysis. com. Yeah. Awesome. Nice way to end off. Thanks guys for joining. Appreciate it. Yeah. Thank you, Jordan. Thank you Jordan.