0:00
You're watching TVN.
Search this video
0:03
Uh, this week, what were the top stories?
0:04
What were the most interesting things that we learned?
0:06
One, secret interviews, the the the Ukraine drone attack. That was huge.
0:10
Uh, Operation Spiderweb, a whole ton of of uh drones were smuggled into Russia uh in uh shipping containers.
0:20
They emerged and uh went out and attacked bombers.
0:24
We had Saurin Monroe Anderson from Nuros on the show to uh to break that down for us.
0:28
We also talked to Connor Love at Lightseed who does a lot of in uh defense tech investing.
0:35
And of course, a couple weeks ago, we had Eric Prince on the show, the founder of Blackwater, and he had kind of predicted that the uh Ukrainian military was perhaps underrated and and we might be seeing like something like this in the future.
0:47
And so, uh that was interesting to see to see play out.
0:49
So, we will take you through those kind of interviews, recap some of those.
0:53
Then obviously we had the absolute meltdown between between President Donald Trump and Elon Musk.
0:57
Uh that unfolded on X and Truth Social.
1:00
The two uh leaders dueling dueling social platform. Exactly.
1:04
Uh and although it's a highly political story, there are big business implications.
1:10
You talking about what's going to happen in space between NASA, uh Boeing, different launch providers. Yeah.
1:17
The implications for Tesla, SpaceX, Neuralink, even the Boring Company, right? There's a lot of stuff.
1:23
Many of Elon's businesses are heavily regulated. Yep.
1:25
And uh the potential impacts are substantial. Yeah.
1:30
Then we also uh had had uh some earlier stage founders on the show.
1:33
Uh Roy from Culie came on and went pretty viral. Put on a show.
1:39
You're surrounded by journalists. Hold your position.
1:40
Clue is a uh is a is a service to help you cheat on everything.
1:45
Um we got to actually try the app.
1:48
someone was asking was was pushing us to try it and see how good the product is.
1:52
Um but regardless of the product, he's also a phenomenal marketer and he came on and put on an absolute show and he's printing apparently. Yeah, he's doing great.
2:01
You can't spend all the money that they're bringing in.
2:04
And so we'll give you an update on Roy and see where that business is.
2:07
And then Keon uh from Nucleus came on to launch Nucleus embryo uh which he calls the first ever genetic optimization software that helps parents give their children the best possible start in life.
2:18
Quite a lot of controversy there on the timeline this week. A lot of people hate it. A lot of people love it.
2:25
Um and uh we'll let you kind of decide for yourselves.
2:30
And then we had a whole bunch of AI experts on the show from Google open AI and anthropic got to the front leading edge of the debates around AI AI LLM. Poor John.
2:41
Yesterday you were fighting as long as you could.
2:43
You didn't want to talk about uh drama. You got dragged into it.
2:49
Uh but we did get some great coverage from Mark Chen at OpenAI as well as Schol over at Anthropic.
2:54
Yeah, it was a lot of fun.
2:57
The big news over the weekend was the Ukraine drone attack on Russia.
2:59
They shipped uh shipping containers into deep into Russia at which point uh drones flew out of the containers and uh hit strategic targets.
3:09
Uh we're going to have two guests on the show today.
3:10
Uh Saurin Monroe Anderson from NOS to talk about that and also Connor Love from Lightseed to talk about uh that and also defense tech investing generally.
3:19
Today we have Saurin uh from Nuros who builds drones and has been to the Ukraine and so we'll bring him into the studio and uh ask him how he's doing. How you doing? There he is. Welcome. Great. How are you guys? We're good.
3:31
We have a new soundboard so expect some wild some wild cards. Wild stuff.
3:34
Could you give us a highle overview of the history of drone warfare in Ukraine because I understand it's been progressing super rapidly uh on both sides and I and it'd be helpful to understand kind of the different stages.
3:50
Do they ever have like predator drones like the global war on terror type of drone or did they jump straight to quadcopter and kind of like leapfrog the technology?
3:58
So, you know, you've had this this uh Russian aggression war in Ukraine since 2014.
4:04
Obviously, the full-scale invasion was 2022, but even during that that period before the full-scale invasion, there was some usage of drones for surveillance and uh dropping explosives.
4:15
These are primarily still like small drones like what you're seeing now.
4:19
Uh but this was not a a proliferated technology.
4:21
Then when the full-scale invasion happened, uh within a few months, the Ukrainians started thinking about all these ways that they could use, you know, inexpensive drone technology to get a an asymmetric advantage.
4:34
And that is where FPV drones started becoming a really really big deal.
4:38
So they pioneered the uh really this idea of, you know, putting an explosive on a racing drone and using that as a precision strike weapon.
4:46
There were instances of this happening in other places, but they really scaled it and they've really refined it.
4:52
And then Russia was was much slower to take it seriously, although now they're they tend to uh in some ways outproduce Ukraine and they have a much, you know, more direct line to China um where most of these components are coming from.
5:05
But since 2022 and FPV is just starting to get used now, it's reached an unbelievable scale.
5:12
Uh it's estimated Ukraine is going to produce 4 and a.
5:14
5 million FPV drones this year.
5:16
And those are ranging from, you know, ones that are this big to 15-in propellers, uh fiber optic controlled drones, many different types and sizes of warheads, um different configurations, and I can talk more about the drones that were used in in Operation Spiderweb as well because those were really interesting.
5:36
But uh what we've seen is just this vast technology landscape um where new clever ideas like fiber optic are you know going to be the the hot thing for a few months and then they sort of just become another tool in the tool belt and it's just this constant arms race. Yeah.
5:48
talk about uh talk about this this attack was was unique in a bunch of different ways, but is this something that had been and to your knowledge or or just you know more generally known to be something that had been attempted multiple times or you know maybe like uh I'm curious to know um yeah kind of the backstory on on this type of attack because it seems you know it's a massive difference to be using this technology way behind enemy lines.
6:20
versus using it, you know, at the front line. Yeah.
6:21
So, primarily FPV drones are used on the front line, say the kind of 30 km band across the zero line.
6:28
What was so unique here is that it was FPV drones, short-range drones being used 4,000 km inside of Russia.
6:37
Um it it was this unbelievable application where, you know, you've seen Ukraine using long range one-way attack drones that are going, you know, 1500 km um to strike targets deep inside of Russia.
6:50
But here, these were small drones actually driven in on on uh trucks basically in the tops of shipping containers.
6:58
Um and I don't know of any uh operations that were similar to this beforehand.
7:04
But I think uh it was not something they wanted to uh give away and the drones were actually operating on cellular.
7:11
They were not operating on local like the the normal low latency local radios you use for FPVS typically.
7:19
Um and so I think you know this is going to be something that a lot of people are going to look at and and see if you have drones that are operating on cellular you can't really tell them apart from cell phones.
7:30
That's really hard to defend against, really hard to detect.
7:32
Uh but now it's going to going to be part of air airbased defense is um thinking about drones that are operating on cellular being piloted from basically anywhere in the world.
7:42
Talk about the Russian response, the immediate response to this incident uh from the footage that I saw and I think most people saw that that uh tracked it.
7:51
It it seemed incredibly challenging to respond to it quickly. Right.
7:56
By the time you could sort of organize a response, a lot of the the core damage had been done.
8:00
Uh what what do you think the the the the question I think that every country is asking themselves now is how do you defend against this type of attack?
8:10
Whether you're at war like Ukraine and and Russia are or you're just you know thinking uh you know long term. Yeah.
8:20
This clearly poses a massive threat to critical infrastructure.
8:23
I mean being blatant the US does not have any defenses in place that would stop this from happening.
8:29
We already know we there's already news stories about drones that are flying over our Air Force bases and we can't do anything about it.
8:36
And I think the only approach here has to be a a multi-layered system where you're looking at all of the different types of electronic warfare and also considering things like satellite communications and cellular communications where you're basically able to turn those off on the flip of the switch, which is a huge uh a huge inconvenience and a huge uh thing to build into the infrastructure.
8:58
But clearly that's going to be required.
9:02
Welcome to the stream, Connor. How are you doing? I'm good. I'm doing all right. Good to be back, guys. Yeah.
9:05
Oh, you got a suit this time. Oh, looking great.
9:10
I wanted to I I won't say I dress up just for you, but you know, I would have taken the suit off far before this, you know, if I wasn't coming on. Fantastic.
9:18
Thanks so much for jumping on.
9:20
Uh, have you been tracking the Ukraine story closely? Uh, any insights there?
9:26
anything in the portfolio that's uh at all relevant in the defense tech world?
9:30
Uh do you expect a response from the US government or guidance or change to any strategies?
9:34
Really any takes on that?
9:36
I mean, first [ __ ] what a what a time to be alive.
9:39
I mean uh you know, I'm sure your your your Twitter feeds and and your group chats were blowing up uh pun intended over the weekend.
9:45
Um I mean, it's pretty crazy.
9:48
I mean, let's be honest, like first, I'm not shocked that that the Ukrainians did this.
9:52
Uh, I mean, the execution seemed to be flawless from what we can pull from from open source intel.
9:58
I do think though, I mean, again, it's not a surprise.
10:00
The the Ukrainians have been mastering drone warfare for the last handful of years.
10:06
And, you know, you want to call it that, you know, they called it spiderweb, like this was their this was their Trojan horse.
10:11
This was their, you know, Israeli beeper.
10:13
Um, and and the outcome is is is is pretty impressive to be honest.
10:15
I mean, but what what what from the outside looking in, like the Russians woke up uh over the weekend and they thought they were getting their $4 timu orders and what did they get?
10:24
They got a thousand, you know, FPV drones, you know, blowing them to smitherine.
10:28
So, it's pretty impressive.
10:30
I mean, my my takeaways from this are really twofold.
10:34
The first is like there's never been a clear signal of where warfare is going.
10:39
Um and to be clear what you know what I what I view this from you know both the the entrepreneurs in my portfolio but also from my perspective I mean the world is about uh you know cheap attraitable a lot of times uh autonomous systems and that's you know playing out in warfare that's playing out in other
10:55
areas of life and then the second thing is um you know candidly it's like uh it's it's really hard to defend yourself at the pace at which things are changing um and and again like I know we do some things here in the United States and trying to be on the front end of a lot of this innovation. But when this
11:10
But when this happens, I think this almost just resets everyone again and says, "All right, how do how do we respond to it?"
11:15
And I think it's to your point, it's not a it's not a direct US response.
11:19
It's more of, hey, what do we need to buy?
11:21
What do we need to develop, you know, for our own fight in, and you know, in some way, shape, or form? Yeah. What do you think?
11:26
Uh, obviously, you're a venture capitalist, not a geopolitical strategist, but what's what what's the right Russian response to this?
11:34
is that hey, we suddenly need to be wary of having cell coverage anywhere near strategic military assets.
11:41
I mean, it seems like Ukraine and Ukraine in Ukraine's perfect world, they could run this style of attack a bunch and copy and paste and hit other targets, but it feels like something that was dependent on cellular technology that that's something that the the Russians can revoke, you know, fairly fairly quickly.
11:59
Sure, it'll be inconvenient, but I'm curious if you have have a take.
12:04
Yeah, I mean I, you know, to be honest, when I think about um how do you defend against this?
12:09
I think there is, you know, I wouldn't call this the easy answer of just, you know, turning off off the cellular network.
12:16
Um, I actually think the the only way to do it kind of practically is in layers or in a multitude of of of different ways because, you know, yeah, the, you know, the reality is if you looked at how the Ukrainians carried out this attack, they did so on the local, you know, Russian cell network.
12:32
Um, which again I don't think any Russian kind of defense unit at any of these bases was ever thinking that they would have to turn off their own cell network.
12:41
And then there's just the practicality of how you do it.
12:42
I mean I think there was what four or five different attacks that hit all at the same time.
12:46
What do you what do you do?
12:48
You you turn off the network for tens of thousands, hundreds of thousands of people.
12:52
And oh by the way, this is like a dirty little secret that nobody talks about.
12:54
You know, yes, you have your military systems that are protected and all that, but a lot of coordination is happening through WhatsApp.
13:01
a lot of coordination on and so all of a sudden you turn off the cell networks, you're actually inhibiting your own defense, your own response, the the first responder, you know, the uh you know, getting your own people out of there.
13:12
So I I think it's a bit more complex than that.
13:14
And then the last thing I'd say is just like uh even if you do this in layers, you know, you you need to be resilient in a way, but you're you're not going to stop everything.
13:24
everything. I mean this was just brilliant master class of you know again if if maybe there was a plan we didn't know this but maybe there's a plan for you know a hundred bases and we only hit five of them and and and so if you think about just the the broad you know
13:37
geopolit you know geographic coverage you you have to have I think to be 100% certain on anything it's just you it's impossible you can't do it the most capable military in in Europe right now is the Ukrainian military the lessons learned that they have are very significant. The drone tech is far and
13:54
The drone tech is far and away the best.
13:57
Their ability to to fight against and to even conduct electronic warfare and even close air support in this environment is uh is leaps and bounds ahead of even what the US military is.
14:11
So that's the military to learn from.
14:14
Uh Ukraine does does have a corruption problem.
14:17
I hope sincerely that Trump is able to to get a ceasefire in place and to stop this killing because it's it's absolutely pointless.
14:26
It's just Slavs killing Slavs at this point and it's it's nobody's going to advance.
14:31
Um and you have a um a blend of old and new.
14:36
I mean, if you look at the pictures of the front there now, it's almost indistinguishable from the Battle of the S, right?
14:44
artillery duels, static lines, bunkers, all the rest.
14:48
Now, the problem is somebody can fly an FPV into your bunker on the other side, but between tens of millions of landmines, which make armored breakthrough very difficult, it slows down any attack so that the FPVs and and artillery can get to it.
15:03
You're not going to see any kind of blitzkrieg, Hanserian maneuver warfare there um until some significantly different weapon systems come along.
15:14
So look, Europe needs to get serious about it.
15:18
They're far from it at this point.
15:20
Elon Post four minutes ago, the Trump tariffs will cause a recession in the second half of this year. Wow.
15:28
Um, somebody else was saying, "Can I finally say that Trump's tariffs are super stupid?"
15:35
Somebody else is posting Mad's posting is saying it's Xi Ping.
15:37
He says, "Bro, you seeing this?"
15:40
And it's um Putin on the other end. He's just looking at it. Hold up. Got a line.
15:44
And it's uh we'll start pulling some of these up. Ridiculous.
15:53
Uh what else is going on here?
15:53
This is the president versus Elon.
15:56
Naval says Elon's stance is principle principled.
16:01
Trump stance is practical.
16:01
Tech needs Republicans for the present.
16:03
Republicans need tech for the future. Drop the tax cuts. Cut some pork. Get the bill through. This is so crazy.
16:13
Antonio Garcia says, "Remember, there's fu money and then there's f the world money."
16:21
Will Stansel says, "Imagine being the ICE agent suiting up for your biggest mission of all time."
16:26
Right now, people are saying that Trump's going to deport Elon.
16:30
Elon back to South Africa.
16:30
Um, Will Depuse says, "Time to drop the really big bomb.
16:36
Growing Daniel is in the Epstein file.
16:37
That is turn into a coffee pasta."
16:40
That is the real reason they Oh, no.
16:44
Oh, no. Uh abs um Delian uh uh we had a question from a friend of the show said the real question is if Tesla is down 14% how could SpaceX and OpenAI be trading if they were how would they be trading if they were public the the real thing here is it's bad for everyone
17:06
right down Trumpcoin is down nobody's really winning here China is up uh Shamagu I mean I'm just saying like at a high you know, China is the big beneficiary here of um Sarah Guo says, "If anyone has some bad news to bury, might I recommend right now?" Yes. Yes. Yes. If you have Yes. Yes. Yes.
17:25
If you have if you uh what what's the canonical bad startup news like, "Oh yeah, you missed earnings or something. Drop it now."
17:37
Inverse Kramer says, "Bill Aman is currently writing the longest post in the history of this app." Okay.
17:44
Um, and we have uh we have a video from Trump here.
17:47
If we want, I can throw it in the tab and and we can share it on this on the stream and and uh react to Lex Freiedman says to Elon that escalated quickly. Triple your security.
18:00
Be safe out there, brother.
18:00
Your work SpaceX, Tesla, XAI, Neurolink is important for the world.
18:05
We need to get Elon on the show today.
18:07
If somebody's listening and can make that happen, I would love to hear from Max Meer says, "So I got this wrong.
18:12
I didn't say it never happened, but I thought it wouldn't.
18:15
I'm floored at the way this has happened. Yeah.
18:16
Uh he didn't think they would have a big breakup.
18:19
Uh many people didn't think they would have a big breakup.
18:22
Even just earlier this week, it seemed like they might just have a a somewhat peaceful exit.
18:26
Um Trump just posted a little bit ago, I don't mind Elon turning against me, but he should have done so months ago.
18:34
This is one of the greatest bills ever presented to Congress.
18:38
It's a record cut in expenses, $1.
18:40
6 $6 trillion in the biggest tax cut ever given.
18:43
If this bill doesn't pass, there will be a 68% tax increase and things far worse than that.
18:50
I didn't create this mess. I'm just here to fix it.
18:52
Um, anyways, lots going on.
18:52
Uh, let's go to this uh this Trump video.
18:58
I want to see what he has that I've seen.
19:01
I'm sure you've seen regarding Elon Musk and your big beautiful bill.
19:03
What's your reaction to that?
19:05
Do you think it in any way hurts passage in the Senate, which of course is your seeking?
19:10
Well, look, you know, I've always liked Elon and it's always very surprised.
19:14
You saw the words he had for me.
19:16
The words of And he hasn't said anything about me that's bad.
19:20
I'd rather have him criticize me than the bill because the bill is incredible.
19:24
Look, Elon and I had a great relationship.
19:28
Uh I don't know if we will anymore.
19:30
I was surprised because you were here.
19:32
Everybody in this room practically was here as we had a wonderful send off.
19:36
He said wonderful things about me. You couldn't have nicer. Said the best thing. He's worn the hat.
19:42
Trump was right about everything.
19:43
And I am right about the great big beautiful bill.
19:46
But I'm very disappointed because Elon knew the inner workings of this bill better than almost anybody sitting here, better than you people.
19:56
He knew everything about it.
19:56
He had no problem with it.
19:58
All of a sudden, he had a problem.
19:59
and he only developed the problem when he found out that we're going to have to cut the EV mandate because that's billions and billions of dollars. And it really is unfair.
20:06
We want to have cars of all types, electric, we want to have electric, but we want to have gasoline, uh, combustion, we want to have different, we want to have hybrids, we want to have all, we want to be able to sell everything.
20:20
He hasn't said bad about me personally, but I'm sure that'll be next.
20:23
But I'm I'm very disappointed in Elon. I've helped Elon a lot. Mr.
20:27
President, did he I just want to clarify.
20:30
Did he raise any of these concerns with you privately before he raised them publicly?
20:33
And this is the guy you put in charge of cutting spending.
20:37
Should people not take him seriously about spending now?
20:39
Are you saying this is all sour grace? No.
20:40
He worked hard and he did a good job.
20:41
And I'll be honest, I think he misses the place.
20:46
I think he got out there and all of a sudden he wasn't in this beautiful oval office and he was and he's got nice offices, too.
20:52
But there's something about this.
20:54
When I was telling the chancellor, folks, this is where it is. People come in. Breaking news.
20:57
Delian as Bruhov is joining us in the temple.
21:00
I love it for some live reactions. Surprise guest.
21:08
I can't even spell surprise guest.
21:10
I'm so excited about this. Surprise.
21:12
In other news, 11 Labs dropped a new product.
21:20
In other news, $2 million seed round. Stop it. Stop it. We love 11 Labs.
21:25
No, they'll they'll keep grinding, but just launch again tomorrow.
21:31
They're going to have to launch again.
21:32
Start shooting a new vibe reel.
21:35
Start shooting a new writing a new blog post cuz no one's going Lulu says yes.
21:39
Delay the launch on TVPN.
21:42
So basically right now I can just pull up and just refresh.
21:46
I'm going to just be refreshing True Social. What are you doing?
21:50
So okay, Jordy's on Truth Social. I'll be on X.
21:52
Give us your reaction, Delian. what's going on?
21:55
I mean, at some point I was like, I'm just, you know, sort of scrolling up cuz I like tuned into you guys like an hour ago and I was like, they're talking about some AI thing.
22:03
I was like, some point switch to like was like and then I was watching it and I was like, okay, like John resisted.
22:08
I fought it for like for like a half an hour.
22:11
Um, but we couldn't do it.
22:14
Um, but yeah, give us your quick reaction.
22:16
I mean, I'll always, you know, sort of give it from the uh, you know, sort of space angle.
22:22
You know, it's amazing that, you know, um, uh, how much the world has shifted since, you know, Friday of last week where it was, you presumed that Jared Isaacman was going to be the, you know, sort of NASA admin to today, um, it was released that the, uh, Senate
22:35
reconciliation package readded, uh, budget back into NASA largely for the SLS program, which was basically the program that, you know, sort of Jared and Elon were, you know, sort of largely um, advocating to, you know, sort of completely shut down. Um so you know the
22:46
Um so you know the the the um it is already show like you know the sort of counter reaction you know is already showing up you know in in policy.
22:57
Sorry SLS program is that space shuttle or no uh sorry that's the SLS uh launch rocket.
23:02
Um it is based off of old space shuttle hardware but it is basically the internal um you know sort of NASA run competitor effectively to like a you starship heavy you know launch rocket.
23:13
Um and so you know because it was you know sort of generally behind budget behind schedule and there are so many commercial heavy lift rockets coming uh online um the default was cancelled that is largely you know sort of a Boeing based program and so you know if you look at you know you know 3 months ago
23:27
you know when um they were announcing the F47 program you know Elon walks into the secretary of the air force's office obviously he'd been you know ranting against um you know man fighter jets and bleeding that shouldn't be what you know uh be what the department is prioritizing 30 minutes after that
23:41
meeting was when they announced the F47 program and so now you're seeing basically like the equivalent in space where you know uh you know that you was obviously awarded to Boeing Boeing was the is the largest prime behind SLS uh you know Boeing basically you know um is going to be the biggest winner of you know NASA refunding IOT SLS and Jared
23:59
Eisman not being NAS administrator so tying this back to the timeline Trump posted less than 30 minutes ago in light of the president's statement about cancellation of my government contract SpaceX will be begin decommissioning its Dragon spacecraft immediately break that down. I mean, that just means that we no
24:14
I mean, that just means that we no longer have a vehicle that can go to the International Space Station.
24:18
We no longer have a vehicle that can bring astronauts up and down.
24:20
Um, you know, we also don't have a vehicle that can de-orbit the International Space Station safely, right?
24:27
That the Dragon was expected to be able to do that.
24:29
So, what that means is, you know, if you guys remember all the memes about stranded, uh, you know, from last year around Boeing Starlininer, um, it now means that the space station, you know, itself is basically, you know, sort of stranded.
24:40
And that's like, you know, one of the government contracts obviously that, you know, SpaceX is involved in.
24:44
Elon, I've heard generally like just wants to shift all things to Starship anyways.
24:48
And so in some ways was probably kind of looking for an excuse to uh, you know, sort of shut down Dragon and refocus energies.
24:51
There's also a part of it where it's like, look, he is like kind of independent in the space world and that, you know, Starlink's um, uh, total topline revenue is going to be passing the NASA budget um, in the next year or two.
25:02
And so in terms of like size of you know state actor that can influence space you know his own company is basically about to become you know as large of an actor as like the entire United States.
25:11
So I don't think there's going to be like a deescalation here like you know my my you know estimation is like on both sides it's going to continue to escalate.
25:20
Um, you know, if we thought that we lived in dynamic times, you know, when Trump got into office, it's going to be even more dynamic when there's like the dynamism will continue until morale improves.
25:31
Elon the center, AOC the progressive populist and Trump the you know, sort of conservative populist and oh man, it's uh on the timeline.
25:43
I mean the I just have so many questions, right?
25:45
How does this impact Golden Dome, right? What's Boeing doing?
25:51
um is is will Golden Dome even be a viable project without SpaceX?
25:54
It's it's I think there's just going to be more resistance probably to working with, you know, sort of upstarts because they would be ones that would probably be more likely to collaborate, you know, sort of with, you know, SpaceX.
26:04
And so, um so I mean it it feels like it feels like Boeing would be a logical beneficiary of this turmoil and yet they're down today.
26:14
They haven't really popped. Oh, really?
26:16
I mean, I'm not obviously, you know, one to give like, you know, public I I I know.
26:19
I'm just trying I'm working through it myself and it's surprising like Tesla to drop and Boeing to pac basically off. Yeah. Yeah.
26:27
That would be the expectation.
26:27
But there there must be something because there there it feels like this is purely interpersonal between Elon and Trump and not it's not like oh Boeing was secretly behind the scenes the whole time lobbying even more effectively.
26:39
It doesn't you got the well where's the tinfoil hat? It's over there.
26:42
Maybe we need a tinfoil hat segment. Who knows?
26:48
But yeah, I mean when you're in Boeing world, it's like, hey, we're only down 1%. Let's go. The coup of the century.
26:54
My question is, has has there ever been a crash out of this magnitude ever in history?
26:59
Well, in internet history when when Elon and Trump became friends, honestly, world scale probably world history equivalent.
27:06
I feel like there was something in like era in the United States where um you know, crashing out used to mean calling up the New York Times and just ranting.
27:17
Now you can just live post like all your reactions and it's just all real time.
27:21
This is like crash are actually intensifying.
27:25
You actually want to be long crash outs over the next social media platform that they own.
27:30
So you know you got to be on both ex and truth social to like stay on top of things. Yeah. Yeah.
27:34
top of things. Yeah. Yeah. I actually did like a deep research report a while back on like has the richest man in America ever been close with the US president going back to like you know was Rockefeller particularly close and and because the narrative was like oh this is like so unprecedented and in fact it is unprecedented in fact oh really interesting I would have guessed that like Rockefeller was close me too
27:56
me too that's what I was going for was like no I imagine this is always this is always close but no I I I think because the president has become more powerful globally your your your your your point about uh you know, mayor of America, dictator of the world, like it becomes increasingly valuable for the richest man to have a close alliance and so it's become more I I don't know exactly how accurate that research was. totally
28:17
totally possible that like behind the scenes Rockefeller was really close to the president at the time and we just didn't write about it in the history books, but there certainly aren't very many anecdotes about the richest man in America going on like had a great AP US history 2050, you know, a dam where you know Elon Musk called the president at the time a potential pedophile.
28:41
Was it a about Epstein island, B about a cave in the Philippines? See, what a mess.
28:46
No, so Pavle had a good post.
28:49
He was quoting the the big bomb uh from Elon.
28:51
He said, "Hypothetical question about the USA's power structure.
28:55
Is the man with the most access to capital more or less powerful than the political head honcho?" Purely hypothetical.
29:00
It's a good uh question to ask.
29:04
good uh question to ask. I mean I think both like uh archetypes have uh grown both in absolute power but also in relative power to the rest of the globe basically since the guilded era right if you think about like the president of the United States in 1925 I'd say pretty darn powerful but
29:24
like there was clear like you know it was a you know sort of multipolar you know sort of world Argentina was pretty darn rich at the time obviously Europe was still you know sort of recovering from World War I but UK was generally you know doing well like it was not you know clear there was you know, sort of huge, you know, outweight effect. And
29:36
And then if you look at probably the, you know, sort of biggest, you know, industries at the time, you I don't think you could claim that even like Standard Oil at its peak, I'd have to go look at the exact numbers, but that like it had the size of budgets relative to like, you know, the like US government in terms of, you know, sort of budgets, right?
29:52
right? versus I feel like now for the first time you both have you know US president extremely extremely powerful and then you have like you know sort of mag seven effectively like the size of you know sort of you know states like they you know their own state governments and then also just more
30:08
bureaucracy more red tape so like I I when I think about the 1920s like Robert Barren it's like it is the it is the you can just do things era and so you want to build a railroad like yeah you might need to get like one rubber stamp but it's not going to be 10 years and tons of lobbying and all this different stuff. So, you can kind of just go uh
30:22
So, you can kind of just go uh you can just go wild.
30:24
You know, it's bad when Kanye is saying, "Bros, please know we love you both so much."
30:29
It's just like the voice the voice of reason is Kanye West. Yes. Yes. Thank you.
30:34
Bring them together and uh you know, form a peace treaty.
30:38
Nikita Beer just added his pronouns back to his bio. Let's go.
30:43
That's He's got a rubber band.
30:43
And Elon's got a rubber band all the way back to, you know, sort of extreme wokeism, straight back to uh, you know, sort of super climate change.
30:49
And somebody's sharing re-sharing the picture of the the Cyber Truck blown up in front of the Trump Tower in Vegas.
30:58
And it's just like this this is in real life.
30:59
It was foretold, but it was a question of like when and of what magnitude, not if. Mhm.
31:08
Um, always bad if if Vladimir Putin is operating to negotiate between uh, President Trump and Elon.
31:15
I think I think a lot of the world is is waiting for Roy Lee's takey and the clearly army.
31:23
That's who we want people have been asking him to get involved with geopolitics. Wow.
31:30
Um I love the uh Shil Moha put up a uh you know sort of meme about Narenda the like prime minister of India.
31:37
Um you know he basically uh copied and pasted the uh Trump truth social post about negotiating peace between India and Pakistan when it wasn't like actually fully negotiated you know posting about you know uh ne negotiating a ceasefire between Elon and Trump.
31:51
Funny thing is like truth social you can just read all of Trump's posts without creating an account.
31:58
It truly shows that like I would think that you would have to make an account to read them all, but they just it's not gated at all.
32:03
This could be the biggest, but you know, they clearly I don't think they care about monetization.
32:10
Um, Bitcoin is actually uh falling alongside falling falling. Um, wow.
32:17
Bitcoin falling, Boeing falling, Tesla falling.
32:18
Who's the biggest winner of the day?
32:20
That's I think it's China. China. Yeah, China. China really sold off.
32:24
It's It's down 3% today at 101K.
32:27
So, still up, but you know. Yeah. Rough.
32:31
Winnie the Pooh just dipping his uh you know, hands in that pot of honey just snacking away watching from the sidelines. Yeah. Let's see. Chinese stocks. US stocks. Chinese. I can't find it.
32:43
Okay, that's probably my commentary on the day, boys. It was fantastic. Thanks for jumping on.
32:47
Thanks for hopping on so quick.
32:50
Next up, we have Roy from Cluey coming back for an update.
32:53
He's hired 50 interns, I think, or something close to it.
32:58
He said they're bringing every intern on.
33:00
They're bringing every intern on.
33:02
We got every intern coming in.
33:02
Well, welcome to the studio, Roy. How are you doing? Oh, let's go. How you doing? There they are.
33:10
I think we're overpowering you.
33:13
Can uh can can you hear us? Yeah. Yeah, we can hear you. Yeah.
33:17
Make sure we're zoomed out all the way so we can see everybody.
33:18
We got a small army here. This is incredible. Uh how big is the team? kick us off.
33:24
How many you got at this point?
33:26
The team is uh 11 full-time plus the interns.
33:28
How many interns you got so far? Interns, bro.
33:30
We're closing in on 50, brother. Let's go. There. That's amazing. Congratulations. What are they all doing?
33:40
How do you manage everything?
33:41
Uh is it just is it is it purely social media?
33:44
Is that what you want them to focus on? Growth. Yeah. Yeah. Growth marketing.
33:48
Like the only goal of the company is get 1 billion eyeballs onto Culie.
33:51
So, we have unrestricted creative freedom and permission to do anything and everything.
33:57
Uh, just just make the company go viral.
34:00
Every single person you see behind you has over a 100,000 followers on some social media platform. 200,000 plus. Wow, that's remarkable. Me too. Me too.
34:09
Now, yeah, you're up there. Oh, yeah.
34:11
Yeah, you probably popped. Uh, uh, what's working?
34:14
What platforms have actually been uh been driving the most growth?
34:17
I mean, I'm sure you've run a lot of tests.
34:19
What have you learned that's uh that you can share?
34:20
Bro, Ben, take it away, bro. Let's hear it.
34:23
UGC has been really good.
34:26
We just hit 10 million views today.
34:27
Um 10 million views on eight days. Wow. There we go.
34:29
Hoping to get 100 million views in the next month.
34:34
What what what platforms specifically are the most fertile ground for uh targeting your specific customer?
34:40
Because you can imagine that uh there's a lot of folks who are AI curious on X, but then there's much broader, more viral audience, more general audience on platforms like Tik Tok, YouTube, Instagram.
34:50
What's working and what is uh is the next next platform that you're going to be focused on? Yeah.
34:55
Well, we're trying to go viral on every platform regardless.
34:58
Um but the main thing right now is Instagram reels. Oh, Instagram reels. In interesting.
35:02
And what is the main value prop that you're hitting people with?
35:06
Is it still the cheat on test thing or have you evolved at all? What? Still interviews? Yeah. Interviews. Okay.
35:12
And uh has there been I This was controversial when you launched it.
35:18
Is it still controversial in the comments? Are you getting flamed?
35:21
Has anyone big dunked on you?
35:21
And has that driven virality?
35:24
Is that actually a net positive?
35:27
In Instagram is not like Twitter.
35:29
Like you could post the craziest [ __ ] on Instagram and they will still not think it's controversial.
35:32
So how to make it controversial?
35:34
Like we have to engage in bait some other way.
35:38
Like it's cheating tool is controversial on Twitter, but on Instagram, you could you could have like a white guy say the N word 10 times and it's still not controversial enough.
35:43
Like like you need crazy [ __ ] on Instagram. That's what we crack.
35:47
Every single person here has like very great viral sense.
35:49
And you watch the reels that do go viral.
35:51
You see there's like ways that we've engaged and beta the videos and this is what we'll keep doing to uh probably a billion views a month is how how long does it take to figure out if an intern is cracked?
36:01
Is it like an hour, two hours?
36:03
How much time do you need?
36:03
For me personally, me personally probably like 10 minutes, but for anybody watching probably would take like one one or two weeks. There we go. There we go.
36:10
Uh, how do you guys think about how do you guys think about product marketing?
36:14
Obviously, you're just going viral everywhere, getting all this attention.
36:17
How do you make sure that it that it uh I love that he's shaking his head on doesn't think about it's not about the product, it's about the attention.
36:28
Attention make anything go viral. Yeah. Um, yeah.
36:31
But but but how do you the side of the street, you know, you uh make some UGC videos, make some Twitter posts, you know, you could sell anything.
36:38
You know, in 2025, product doesn't matter.
36:41
You know, I could jack off off the side of a building, sell some videos of it for 20 bucks each, make $2 trillion. It's crazy. Two trillion. That That's intense.
36:48
Um how do you guys think about burn? Uh is it on your mind?
36:52
Is it on your mind at all?
36:52
I don't know if you saw the last tweet, but as of literally like two days ago, we're still we're still cash flow positive.
36:59
We're still [ __ ] profitable. We're so profitable.
37:00
Let's give it up for the property owner. Let's hear it. Let's hear it.
37:03
Uh it sounds uh so you're charging for the product and people are paying.
37:07
Are they at all satisfied or do they feel uh like they got satisfied, bro? Like the product works.
37:13
You're either using this as a consumer and it's working cuz like like you're passing your interviews and or if it doesn't work, you're not going to complain to me cuz I'm going to go write to your employer and tell them, "Yo, guess who's complaining about using the product?"
37:24
Like I will get you blacklisted if you complain really.
37:28
Um, where how are you thinking about how do you how are you guys thinking about product evolutions?
37:31
What do you want to add to the product?
37:32
Obviously, uh, you want to help people cheat on everything.
37:36
Where where are you going to help people cheat next?
37:37
We don't care about like the product is going to be led by the virality of the content.
37:42
We have video ideas right now that we're going to try to push for different use cases.
37:46
We're going to see which ones go consistently the most viral.
37:49
If you can make something go more viral, then like you can just build the technology after you have all the attention.
37:54
So, we'll figure out the exact use cases and exact niches we're going to quintuple down on once once these guys get to work.
38:00
What What uh what formats on Instagram reels are like the most modern in terms of uh consistently viral?
38:07
Like you mentioned like man on the street interviews.
38:09
What do you do for a living?
38:10
That's always been a fertile ground.
38:12
What about uh I see a lot of those like mobile game ads that look like, you know, you're fighting down some sort of bridge and then you go into the game, it's actually just uh match three.
38:21
Um what what what are the different formats that you like to pull from?
38:27
Every week there's two new ones and at any point there's probably 10 to 20 viral trends that is happening and these cycle so quick.
38:34
You need to keep your finger on the pulse these things will like expire immediately.
38:38
You need to be on the ball and like like if I if I told you right now by the time people watch this on YouTube like it would have all been expired.
38:44
Well, we're live so so give us give us the latest and greatest like what's going viral today.
38:50
Well, right now we got 10 million 10 million views using a Snapchat format viral for like the last three years to be honest. Okay.
38:58
And and I I I think that like we just have to get people who continuously scroll TikTok like six hours a day. Yeah.
39:05
But what's the what's the actual format that you use? Like describe the video. What is the hook?
39:12
Like break it down for me like you're explaining the like the art behind the viral format. There's a caption.
39:19
caption. It starts with a face usually a handsome dude or a pretty girl is saying damn this interview starting with the interview was starting with the hard questions I should have been a CS major not a business major that's engagement debate because people are saying like bro like CS is way harder than business then it turns around the interviewer asks like like hey how are you doing why
39:36
should we hire you and then this guy uses cluey to generate a response but he can't [ __ ] read the response so he reads it hella autistically like oh I revel in detail and and then that's that is like another conversation point like people are cooking on the guy because he uh he can't read properly a guy is like a doing a really dumb interview using that's great. How are
39:52
How are you guys using uh AI generated content internally?
39:58
I know a lot of these the videos that you guys are creating are just typical social media vertical video.
40:04
Do you have an intern that's just generating uh basically copy and pasting making a bunch of other V3 or any of these tools relev anything clicking? Not not yet.
40:13
I think there's still like a 10% left before they cross the uncanny valley.
40:16
And the biggest thing is that people need to think your video is real.
40:21
That like like that is the difference between 100k views and 10 million views if people think that it's real. AI CEO is bearish on AI. What about uh Yeah.
40:30
What Google needs like 10 more Chinese researchers to like figure it out and once once they push out the latest update then then then VO3 will be there.
40:39
But right now we need real people. Yeah. Yeah. Yeah.
40:40
Uh well I I mean what about just using AI as like stock footage replacement?
40:46
Not not as the leadin for the video, not the entire video, but just like sprinkled in to illustrate a point, you know, an establishing shot of like a building, a helicopter pulling into a building like that that historically has been kind of something that you would reach to uh you know Adobe stock video for.
41:03
V3 feels like it's there, but are you not drawing on that at all yet?
41:08
If there's a viral format when we need it, maybe we'll use it.
41:13
But right now, like it's it's really brain dead to go viral on Instagram. Yeah. Formats are not hard.
41:17
You don't need a helicopter.
41:19
You need a guy, a camera, a really shitty camera. You need a computer. That's it.
41:22
I mean, what about like those those kind of like AI mashups like Harry Potter Balenciaga or the uh the kangaroo with the plane ticket getting on the plane like like AI content can go viral when it's really uh when it's like inspired almost by a human.
41:38
It's not it's not entirely AI generated, but it's using the tools effectively to create something that's like still catchy.
41:43
Uh do you think you'll be using any of that anytime soon? Probably very soon. We're scaling up.
41:49
Like what you see right now is probably about less than 1% of what the size we will be by the end of the summer. Like we are profitable.
41:54
We're not trying to be profitable.
41:56
We just keep making so much money we can't help it.
41:59
So we're literally scaling this [ __ ] up to I'm not even trolling you.
42:01
1,000 creators are going to be shipping out content.
42:05
We're doing a complete internet takeover. Okay.
42:06
So, so, so, so why in house? Why? Like, like like what?
42:09
Why do they even have to be employees? What?
42:13
Couldn't you turn this into like a multi-level marketing scheme or something?
42:15
A pyramid scheme or I knew what we're going to do this again.
42:16
Oh, that's what you're going to do. Okay. MLM MLM.
42:22
Are you guys worried that you could be infiltrated by journalists?
42:25
I'm sure they're circling the house right now.
42:28
The hit pieces are going to come.
42:28
You know, we're we're doing a softball interview right now.
42:31
I mean, the the person that's brave enough to try to do a hit piece on the Culie Army is I bet they're dying, too.
42:38
Look, more more eyeballs is better.
42:40
There's no companies that ever died from a founder being too controversial.
42:45
You got deal [ __ ] infiltrating with genuine spies and they're still doing fine, bro. You got worker 17 guys.
42:50
They're they're still kicking.
42:53
Like, no company ever dies from being too controversial.
42:55
You die because you don't make enough [ __ ] money. Yeah. Yeah. Yeah. Yeah.
43:00
Uh, speaking of making money, uh, what's the pricing model right now?
43:02
Are you doing anything on price discrimination?
43:05
Is there a super high tier if you get a whale?
43:07
What does a clearly whale look like?
43:09
Can I spend $2,000 a month on this service?
43:11
Yeah, you should add a tipping feature, too.
43:13
People should be able to tip you guys if if they have a good experience to get the job.
43:16
Like really financialization, pay as you go, high interest rate loans, just really push it.
43:22
Make it sports gambling in there.
43:23
Maybe just throw it all in. Yeah.
43:23
I mean, it's $20 a month for consumer, $100 a year.
43:27
and and our our topline revenue is really being driven up by enterprise [Music] enterp what whatever what are the is that more on the sales side what who are the enterprise so you sell the STRs you guys laugh cuz you think I can't sell enterprise cuz I'm No I I don't believe it like these 20 these Fortune 500 CEOs like these are like 35 year old dudes who sit there scrolling through Twitter laughing at my post Yeah Oh, yeah. Yeah.
43:58
No, no, it seems seems legit. It makes sense. No, I I I believe it.
44:00
But, but I mean, you're not going even higher tier like with the $2,000 a month clearly vision for for consumer.
44:06
There's a lot more we can do with more compute, but right now we're like to be honest, I didn't expect to grow this fast.
44:13
The edge team is quite small.
44:14
I'm like spend a lot of time trying to hire more more competent engineers.
44:17
We have a lot of backlog tasks that we need to fill out, especially for this last contract that we signed.
44:22
So, we're full-time focusing on the one big guy that we got right now.
44:25
And after that um the then we'll we'll try and scale this up.
44:28
But right now we're focused on the one one big client that we signed. Yeah.
44:31
Uh talk about your compensation strategy. The people want to know.
44:35
Uh you you said uh you can raise infinite capital and you're so confident. I believe you.
44:40
Uh but but I'm curious to to get some more insight there.
44:44
Bro, I feel like it's so [ __ ] to be a company. Sorry.
44:47
Am I allowed to say that? No, you're not allowed.
44:50
No, this is a family-friendly show.
44:51
It's very stupid to be a company like trying to race to the bottom to see how little you can pay your employees.
44:57
Bro, if I'm making hella money, we're all making hella money.
44:59
Like, like it's I'm trying to pay them more to see if man like maybe tomorrow we'll start being like cash flow negative, but make money, bro.
45:08
Like, I I would like to pay these guys what they're worth and the output is [ __ ] insane.
45:11
We did 10 million UGC views in what, like 8 days?
45:15
Like like you don't see this sort of traction in any company and you don't see killers like this in any company unless you paying these [ __ ] like what they're worth, bro.
45:21
Like I don't know what like maxed out contracts. Maxed out contracts. Exactly. Exactly. Yeah. Uh what about devices?
45:27
I mean it seemed like this would be a natural fit for some sort of AI wearable or other platform.
45:33
Um is there an app coming or are you interested in what's happening with Johnny IV and OpenAI?
45:41
What's what's your take on the device world?
45:43
We're very interested in the hardware space.
45:45
We've got like a million things cooking on hardware.
45:47
We got people in the garage right now working on [ __ ] you don't you don't even know about, bro.
45:53
Like like like we're we're bringing manufacturing back to America and it all starts at the Culie garage. Let's go. I love to see it.
46:02
Nobody, you know, they doubted, but you guys are re-industrializing America.
46:03
You guys really are the brain chips down there.
46:07
They're working on brain chips down there. Brain chips. That's the future. There we go. There we go. the new Neurolink.
46:13
Yeah, I mean I I you know there's a world in the future where you guys actually just roll up Neurolink and OpenAI bring the fluy umbrella, right? Definitely. It's possible.
46:20
I'm excited to offer acquisitions for for both of those companies. It's in the road map. It's on the road map. All right.
46:26
This has been a lot of fun.
46:28
I'm excited for you guys.
46:30
Uh it is uh and uh I have no doubt that you'll go from, you know, 10 million views a week, 10 million views a week to 100.
46:37
Uh, and I'm excited to see you guys hit that billion uh, view mark very soon. So, keep it up.
46:42
We are all very entertained and uh, rooting for you. Sh. I love the energy. Thanks, man.
46:48
We appreciate you joining. Later, guys. Keep having fun. Bye.
46:55
Next up, we have Keon from Nucleus coming on uh, with a big announcement.
46:59
Something like 10 years in the making, close to it, maybe seven years. Uh, we'll bring Keon in.
47:02
Let's play some soundboard. How you doing? Welcome to the show. That's a great intro. The tweets are flying. Oh. Oh my god.
47:12
Are you guys seeing this? You seeing this? Seeing this. Break it down for us.
47:17
Explain what's happening.
47:21
There's nothing like a launch day.
47:22
I'm trying to figure out guys.
47:23
Is this is it Datakica or is it Therronos?
47:26
Cuz people can't they can't make up their mind.
47:28
Oh yeah, we're going to find out. Trying to figure it out.
47:31
They're trying to figure out what's going on.
47:32
Let's give some context to the audience.
47:33
Nucleus has launched Nucleus Embryo, the world's first genetic optimization software.
47:37
Basically, parents can, you know, give their children the best start in life.
47:41
They can pick their embryo based off of physical characteristics like eye color, IQ, they can go to disease risk like cancers or heart disease.
47:47
Basically, we really believe parents can get old information that exists about their embryos and they can pick however they want.
47:55
Um, for me personally, you know, it's been 10 years in the making.
47:57
The journalist actually covered it today in the Wall Street Journal was a journalist that covered my gene editing in a warehouse in Brooklyn 10 years ago. Yes. Let's see. Wow. Overnight success.
48:06
It's you know it's a long time in genetics. Yeah.
48:11
So so so break down the state-of-the-art because like embryo screening exists.
48:14
I think most parents in America at least if they the means do some sort of screening uh while the embryo is growing. Is this purely for IVF?
48:24
Is this just going a layer deeper?
48:27
And then is I want to talk about the regulatory and FDA component as well.
48:32
Yeah, let's talk about it.
48:32
So, oh, basically if you go to IVF clinic today, you're a couple.
48:35
The vast vast vast majority of clinics.
48:37
The first thing I have to understand is that the IVF process is principally controlled today by clinicians or doctors.
48:42
Honestly, couples don't have as much liberty in our perspective as they should. It's their baby. It's their embryos.
48:47
They should have the right to those that information and they should be able to pick off any vertical.
48:51
However, today in the clinic, what generally happens is people test embryos for very rare and severe genetic conditions.
48:57
For example, like a chromosomal abnormality like Down syndrome, for example, or even a condition like cystic fibrosis or T-ax or PKU, right?
49:05
These are conditions that are very rare um that maybe someone might have a carrier for cystic fibrosis, but it's it's pretty rare.
49:11
Um then there are conditions that we've all heard heard about things like breast cancer, things like coronary artery disease, the things that actually kill the vast majority of people today, right?
49:20
chronic conditions kill the vast majority of people today.
49:21
Those conditions are just not tested for in the clinic even though we have very good science actually that can make those predictions.
49:27
How do we know this as a as a DNA companies?
49:28
Well, that's what we do, right?
49:30
We build models that predict disease and the way you test those models in adults.
49:33
So, we go from adults to embryos is actually because we can basically well validate these models to show that they work in both the embryionic context and in the adult context.
49:41
And so, what we're really doing is we're going from okay, instead of just looking for really severe like down syndrome cystic fibrosis, why not do breast cancer?
49:48
Why not do heart disease?
49:50
Why not do colurectal cancer? Right?
49:50
Why not do schizophrenia? Why do Parkinson's? But then why stop there?
49:55
And this is really the important thing because ultimately, you know, if you think about diseases and traits, the extreme version of any uh trait is actually a disease, right?
50:02
Height is a good example of this.
50:04
One extreme end is like, you know, John for example, he's like Mark syndrome.
50:07
Then the other end is like me, dwarfism, right? On both ends. Okay?
50:11
So, you know, so you know, IQ is another example of this.
50:14
One end is like, you know, autism.
50:16
is like, you know, autism. the other end it can actually be some sort of you know a cognitive basically challenge that people have and so when you think about it when we start realizing that people have drawn a line in the sand saying you can get you know rare diseases you can't get common diseases but then they really said you can't can't get any traits like
50:33
height even though the best predictor we have today actually in the world the best polygenic predictor is for height so as a company we've kind of completely reimagined this and said wait a second what's going on here you should have access to the entire stack rare diseases we do cystic fibrosis common diseases like breast cancer and also traits all the way up to something like IQ. Yeah. Yeah.
50:51
So I mean uh that test are you just giving people the data because I imagine that once you get into particular recommendations that's more what I would expect a a licensed doctor to need to do.
51:05
Well yeah my sense is that they you can allow people to get the data from their doctor and then and then feed it into Nucleus. Is that correct?
51:11
So that's that is correct.
51:13
And actually we have a couple there was like 10 announcements today. You know how we do it.
51:16
We like to do 10 announcements in one day.
51:17
We are actually very very excited to announce uh a huge partnership with Genomic Prediction.
51:23
So Genomic Prediction is actually the oldest uh embryo testing uh company that exists.
51:27
They've done genomewide testing embryos for almost a decade at this point.
51:30
And I think they've done over 120,000 couples for PGTA which is specific kind of test.
51:34
And so we're actually partnering with them.
51:37
So we make it very easy for genomic prediction customers to request their files and actually port it over to Nucleus.
51:42
But really this isn't just for genomic prediction customers.
51:44
Anyone who's undergoing IVF can go to their clinic and say I want my embryo's data.
51:51
You can take that data you can upload it to nucleus and then all of a sudden you know the application of DNA makes this technology universally uh basically universally accessible.
51:59
Now, how much of how much of the benefit is is actual uh algorithmic analysis bringing in other data points to contextualize the data versus just better UI and better hydration of existing text because we we uh we we had we had a friend on the show who was talking about um getting some medical results from a doctor.
52:21
The doctor's office was closed.
52:22
It took two days to until the doctor was going to be able to interpret the results.
52:27
he was able to just take a photo, upload it to ChatGpt and say, "Hey, is this is this a you know, is this really really bad?
52:35
Should I be panicking because it seems somewhat out of the range?"
52:37
And Chat GPT was able to say, "Hey, you still got to talk to the doctor, but this isn't this isn't the craziest thing I've ever seen.
52:43
This isn't way out of distribution."
52:43
And so that's almost like a pure UI layer, but extremely valuable.
52:47
I know it might not be like the right narrative for some people that it's like not as innovative, but I think that like all that matters at the end of the day is giving people benefits, right? always both. It's always both. You you you Yeah.
52:59
fundamentally technology just for technologies sake is not siliconized about right about making something that people want. Okay.
53:04
And people can actually use.
53:06
So you think about the the nucleus innovation. It's it's it's twoprong. Okay.
53:08
One is in the informatics, right?
53:10
You know, I've been doing this for 5 years.
53:11
I I almost I would argue to myself that I probably spent too much time, you know, developing the science, right?
53:16
Science in in it's in a nutshell isn't actually very useful.
53:19
You need expanded access to it.
53:21
on multiple different kinds of analyses that make it such that we can actually provide the most comprehensive analyses that exist today.
53:28
But moreover, and this is really I think a key point to your point, John, is people understand them. People can see them.
53:33
I mean, you can pull up the platform.
53:34
I'm not sure if you guys have shown it already, but it's very easy to sort compare your can actually name your embryos.
53:41
You can stack rank your embryos.
53:43
You can understand what the score means.
53:45
We lead with overall risk or we tell you for example instead of saying you're in the 99 percentile for genetic risk for your condition which you know what does that actually mean?
53:52
We say hey you have you know 5% chance or the like of let's say schizophrenia or some other condition.
53:56
In other words by even overall risk people have much greater intuitive understanding of the results we're communicating to them.
54:00
We have genetic counselors on hand.
54:01
So this really is a wait what are we showing here?
54:03
We showed something we showed yeah we pulled up your website.
54:07
That's another thing that's that's a fun one. That's an Easter egg. That's an Easter egg.
54:09
That that's the that's the kind of approach that we're taking here and I think consumers are responding to it, right?
54:17
People want to have access to their data.
54:18
The clinician, the doctor shouldn't decide what ambient implant you should. Okay.
54:22
So, talk to me about what requires FDA approval.
54:23
Obviously, uh new medical devices like if you were developing a machine to take in an embryo and sequence the DNA, I would expect that the FDA would want uh an approval for that medical device.
54:35
But if you are taking data and just showing it to a customer in a different UI, that feels like probably a very light FDA process.
54:46
And there's probably a continuum in the middle where once you're making a recommendation, they they have rules around that, right?
54:51
We as a company do not tell you which embryo to implant.
54:56
You know, basically parents, the couple has complete agency to decide how they want to use the information to implant their embryo.
55:03
Moreover, let's be clear, height, right?
55:06
I mean is can a height analysis be a medical device?
55:08
medical device? you know that doesn't even make sense right IQ height these traits for example we all you know traits are something that I don't think necessarily belongs in even the kind of infrastructure of think about medical care right these are things that go beyond medical care these are things like you know that that people just kind of intuitively know and that there are
55:23
DNA tests done every single day Q to see for these analyses because they're not disease analyses right so we do both diseases and traits to be clear my point is many of these innovations that you have to wonder like you know should should the government say if someone can or cannot pick their embryo based off height that doesn't seem right to me I think it should the complete liberty of of the individual to decide that. Yeah. Yeah.
55:42
But I mean, we're we're a democratic country and so if if you know a huge swath of the population says that the FDA should review that type of uh test or that type of analysis, analysis, it could happen.
55:55
I mean the the FDA reviews all sorts of different stuff.
55:59
And so I I guess the question shifts to like do you expect a change from FDA on uh the way these uh these analysis tools are regulated?
56:12
Um I think right now the most important thing is just putting these highquality rigorous scientific results in people's hands and helping them basically have healthier children, helping them give their child the best start in life. Yeah.
56:23
Um, you know, I I think that generally speaking that, you know, people should have more liberty, more choice in in medicine.
56:29
I think the broader longevity trend actually touches on that point as well.
56:33
Um, so that's what we're excited to do at Nucleus. Yeah.
56:36
I mean, the the fact that you're partnering with a company on on actual on the actual like medical device side, like they are doing the sequencing of the embryos, like that really takes it out of the therronos question entirely.
56:47
In my mind, I think feel like you should be beating the drum there a little bit more.
56:50
It's like like we didn't say we created some new device, but I don't know.
56:54
You have to find We ship. That's the difference. We ship. It's locking, baby. Go look at it. Go use it.
56:57
That's That's the evidence. Why is there a footing? Okay.
57:04
Um I love the visual of John and his wife selecting between embryos and it's like 610 or 72. Tough choice.
57:14
Well, if we go with the 610, uh, he has, you know, potentially flying commercial once in his life.
57:18
We can actually, we can actually play this game right now.
57:21
Okay, here we're going to play a game right now.
57:23
I'm going to put in the chat. Pick your embryo. com. Okay, everyone listen. com. I'm going to go to it. Oh my god. Here we go. Little Easter egg here.
57:32
Okay, let's see what's more important to you, John.
57:35
Intelligence or muscle strength? Come on. Oh, absolutely. Muscle strength. Let's go.
57:38
We're the future bodybuilding. Let's go. Okay.
57:40
John would John would take a he would he would happily have a 52 son if he had you know top 001% bodybuilding gym. Exactly. Yeah. Okay. Lifan or height? Uh come on. Lifespan. Come on. Lifespan. Let's go. Let's go. Let's go. Let's go. Maybe low depression.
57:59
You got to be golden retriever mode.
58:00
You got to be uh You need low depression. You need low depression. Let's go low odd.
58:04
I don't mind bouncing around a bunch. Okay. What's anxiety? Uh, let's go. High risk takingaking. There we go. Got Okay. Wait.
58:14
This is some generative stuff going on. This is great. I got Nadia, too. Enduring athlete. Let's go.
58:22
Physically strong, cautious, built to last. Yeah, this is great.
58:24
Is this driving a lot of uh a lot of attention? A lot of downloads. Is this going viral yet?
58:30
This seems like something that's designed to be sharable.
58:32
We just dropped it right now.
58:34
Technology Brothers, we got you the exclusive. Let's go. There you go. Let's put it out there.
58:36
You know, they can pick your embryo.
58:38
People say, "What's it like?
58:39
Maybe you're not doing IVF yet. No problem.
58:41
Only only 9% of people choose Nadia. Okay.
58:44
Well, we're we're we're contrarian. We like that here. That's fun. Yeah, it's great. Oh, well.
58:48
Well, uh congratulations on the news.
58:53
Congratulations on the launch.
58:53
Um yeah, the pace is wild.
58:55
Last last thing, what's going on with uh Have you seen these just blood billboards? Oh, yeah. They're all over LA.
59:00
So, so there is there is someone who's running a campaign right now, Justice for Elizabeth Holmes, claiming that Theronos was not the scam people think it was.
59:12
And there's a there's a documentary coming out and there's billboards all over LA for just blood. Like, it's just blood.
59:18
It's not that big of a deal.
59:18
And John, to be clear, there's an exclusive on Technology Brothers next week about from this person, right?
59:23
They they're going to tell their story next week.
59:26
Just to make sure you you invited them already.
59:27
We're we're we are we are toying with the idea that someone reached out to kind of connect us.
59:32
We're we're thinking about doing it, but we're not we're not 100% sure that it would be appropriate for the based on the website.
59:38
I don't know if it I don't know if it's appropriate.
59:40
Yeah, it doesn't look like it was designed with Figma. So, I don't know. We can't quite do it.
59:45
It's a little bit The team definitely doesn't use linear. Yeah.
59:47
But they they they claim that uh that Elizabeth Holmes has been proven innocent.
59:52
And so, it's a bold claim.
59:53
We like we like to see people making bold claims by what uh jury is my question.
59:59
Yeah, jury of someone who knows HTML.
1:00:02
Keon, uh, always a great time. Energy is fantastic. Electric. Electric.
1:00:07
Thank you for coming on, firing us up.
1:00:09
Congratulations on the launch.
1:00:12
We will talk to you soon on Twitter for sure.
1:00:14
Okay, so we'll see you there. Bye.
1:00:15
We have someone from OpenAI here.
1:00:19
We're going to stick to technology and business, but welcome to the show, Mark Chen. Good to see you. Great to see you guys.
1:00:26
Thanks for having awkward day, but I'm excited to talk about deep research.
1:00:29
I am excited to talk about AI products.
1:00:31
Uh would you mind introducing yourself and kind of explaining what you do because OpenAI is such a large company now and there's so many different organizations.
1:00:38
I'd love to know uh how you interact with the product and the research side and anything else you can give to contextualize this conversation. Yeah, absolutely.
1:00:47
So, first off, you know, thanks for having me on. You know, um I'm Mark.
1:00:50
I am the chief research officer at OpenAI.
1:00:52
So, uh in practice, what that means is I work with our chief scientist, Jacob.
1:00:57
And you know we set the vision for the research or we set the pace.
1:01:01
We hold the research or accountable for execution.
1:01:03
And uh ultimately we really just want to deliver these capabilities to everyone. That's amazing.
1:01:09
In terms of research, I feel like a lot of the what happens in the research side is actually gated by compute.
1:01:15
Is that a different team?
1:01:17
Because what if the researchers ask for a 500 billion dollar data center?
1:01:20
Uh that feels like maybe a bigger a bigger task.
1:01:24
So yeah it is useful for us to factor the problem of uh research and also kind of building up the capacity to do that research.
1:01:31
So we have a different team uh Greg leads that um which really thinks holistically about you know data center bring up and how to get the most compute for us and of course uh when it comes to allocating that compute for research uh you know Jakob and myself do that. That's great.
1:01:44
Um and so uh what uh what can you share that's top of mind right now on the research side?
1:01:50
There's been this discussion of pre-training scaling wall potentially the importance of reinforcement learning uh reasoning.
1:02:01
There's so many different areas to go into what's actually driving the most conversations internally right now. Yeah, absolutely.
1:02:08
So, um I think really it's a really exciting time to do research.
1:02:12
Um I would say versus two or three years ago, I think people were trying to build this very big scaling machine. Yeah.
1:02:19
Um, and really the reasoning paradigm changed a lot of that, right?
1:02:24
You know, like reasoning is really taking off and it really opens this new playing ground, right?
1:02:27
It's like there are a lot of kind of known unknowns and also unknown unknowns that, you know, we're all trying to figure out.
1:02:35
It kind of feels like GPT2 era, right?
1:02:37
Where there's so many different hyperparameters you're trying to figure out.
1:02:40
And then I think also, you know, um, like you mentioned, you know, pre-training, that's not to be forgotten either.
1:02:45
either. um you know today we're in a very different regime of pre-trending than we used to be right um today uh we can't treat data as this infinite resource and I think a lot of academic studies you know they've always kind of treated you know you have some kind of finite compute but infinite data I don't
1:03:04
think there's much study of you know like uh you know finite data and infinite compute and I think you know uh that also leads to a very rich playground for research do we need kind of a revision to the bitter lesson Is that a a reputation of the bitter lesson or or do we just need to re rethink what the definition of of scaling laws looks like? Uh no, I I
1:03:25
Uh no, I I don't think of uh anything as a reputation of of the bitter really like our company is grounded in we want simple ideas that scale.
1:03:34
I think RL is an embodiment of that.
1:03:36
I think pre-training is an embodiment of that and really at every single scale we face some kind of difficulty of this form.
1:03:43
It's just like you got to find some innovation that gets you past the next bottleneck and this doesn't feel fundamentally very different from that. Mhm.
1:03:50
Um what is uh what's most important right now on the actual uh compute side?
1:03:58
Uh we heard from Nvidia earnings that uh that we didn't get a ton of guidance on the shift from uh training to inference usage of Nvidia GPUs, but it feels like it must be coming.
1:04:09
it feels like this inference wave is is is happening.
1:04:11
Uh are those even the right buckets to be thinking about tracking metrics in terms of the the the story of artificial intelligence?
1:04:24
Because yeah, I mean it's like if if the reasoning tokens are inference tokens and and but they're what lead to higher intelligent more intelligent models like it's almost back in the training bucket again.
1:04:35
um what bucket should we be thinking about and and uh and or or are we how firmly are we in the the the uh the the applied AI era versus the research era?
1:04:47
Well, I think research is here to stay and it's for all the reasons I mentioned above, right?
1:04:54
It's such a like a rich time to be doing research, but I do think, you know, inference is going to be increasingly important as well, right?
1:05:02
It's such a core part of RL um that you're doing rollouts and I think you know we see 2025 as this year of agents right um we think of it as a year where models are going to do a lot more autonomous work you can let them kind of be unsupervised for much longer periods
1:05:18
of time and um that is just going to put big demands on inference right when you think about kind of our overall vision right we we lay it out as a series of steps and levels on the way to AGI right and I think the pinnacle really that last level is organizational AI, right? Like you can imagine a bunch of AIs all
1:05:35
Like you can imagine a bunch of AIs all interacting.
1:05:37
Um, and yeah, I think that's just going to put huge demands on inference, right?
1:05:42
On that on that organizational question, I I remember reading uh AI 2027 and one of the things that they proposed was that the AIS would actually like literally be talking to each other in Slack.
1:05:53
Um, does that seem like does that seem like the way you imagine agents playing out like using the to the same tools as humans instead of one agent says, "I'm going to go talk with Teams.
1:06:06
I'm going to talk with Slack and I'm going to do a little negotiating, but maybe it just happens super super fast 247 or or is there like a new machine language that emerges?" Yeah.
1:06:17
Um I mean I think one thing that's really helped us so far in AI development is uh to come in with some priors for um you know how humans do things and that's actually um you know if you bake those priors in they they typically are great starting points.
1:06:30
So I could imagine like maybe you start with something that's Slack like and give it enough flexibility that it can kind of develop beyond that and really figure out the way that's most effective for it to communicate.
1:06:41
for it to communicate. Um one important thing though is uh you know we want interpretability too right I think it's it's very helpful for us today that what the agents do is you know uh easy for us to read and interpret and I don't think you want that to go away as well so I think there's a lot of benefits just
1:06:59
even from a pure like debug the whole system perspective but just let the models you know speak in a way that is familiar with us and you know you could also imagine like we might want to plug in to the system too right so you um whatever interfaces we're familiar with, we would ideally like our model to be familiar with as well. Yeah. Um I think Yeah.
1:07:17
Um I think it's also pretty compatible with uh you know, we uh hit a big milestone.
1:07:23
We got uh I think 3 million uh paying business users fairly recently. Let's go. Yeah. There we go. Let's go.
1:07:34
And um I think uh three gong hits for 3 million.
1:07:38
The gong will keep ringing for a while. Sorry, we had to do it.
1:07:43
I was hoping you would drop a number. Yeah. Yeah.
1:07:47
Um I knew congratulations.
1:07:49
That's that's actually huge. That's amazing. Yeah. Yeah. Yeah.
1:07:51
Um but I think one big part of that is, you know, we have we have connectors now, right?
1:07:55
Um we're connecting into you know like G drives and I think um yeah, you can imagine, you know, like Slack integrations, things like that.
1:08:01
Uh I think we just want the models to be familiar with the ways we communicate and and get information. Yeah.
1:08:06
Uh, can you talk about benchmarking?
1:08:08
It feels like we're potentially Yeah.
1:08:10
Do you think about benchmarks at all? Oh, yeah. A lot.
1:08:14
I mean, but I think it's a difficult time for benchmarks, right?
1:08:19
Um, I think we used to be in this world um where you have these human written benchmarks for other humans, right?
1:08:25
And I think we all have these norms for like what are good benchmarks, right?
1:08:29
Like we've all taken the SAT, we all have like a good conception of what it means to get, you know, whatever score on that.
1:08:35
that. Um but I think the problem is the models are already at the point where for even the hardest human written benchmarks for other humans um it's really near saturated or saturated right um I think one clear example here is the Amy like probably the hardest
1:08:54
autogradable like uh human math eval at at least in the in the US um and yeah the models are consistently getting like 90 plus% on these and so what that what what that means is I think there's um a kind of two different things that people are doing, right? They're they're
1:09:11
They're they're developing kind of model uh based benchmarks, right?
1:09:16
They're not kind of things that we would give to an ordinary human, things like humanity's last exam, things like, you know, Epic AI that are really really at the at the frontier of what what people can do.
1:09:25
Um and I think um the hard thing is it's not grounded in intuition anymore, right?
1:09:30
Like uh you know, you don't have a lot of people who have taken these exams.
1:09:33
So it it makes it harder to kind of calibrate on whether this is a good exam or not.
1:09:38
Um, one of the exciting things that's on the flip side of that is I really do think we're at the era where models are going to start innovating, right?
1:09:47
Because I think once you've passed the last kind of like the hardest human written exams, that's kind of at the edge of innovation.
1:09:53
And I I think you already see that with the models, right?
1:09:55
Like they're helping to write parts of papers. Mhm.
1:09:58
Um and and I think the other kind of way that uh people have shifted is you know there's these you know ultra frontier evals but there also people kind of just indexing on real world impact right you look at your revenue kind of the value you deliver to users um and I think that's ultimately what we care about.
1:10:17
Can you can you uh bring that back to interpretability research like with these super super hard uh math evals for example?
1:10:25
Uh if are are we doing the right research to understand if the thought process mirrors not just not just oneshotting the answer, oh you you you memorized it or you magically got it correct, but you actually took the correct path.
1:10:41
Kind of like you know you're graded for your work, not just the answer if you're in grade school.
1:10:46
Um and and you know Daario said that uh interpret interpret interpretability research will actually contribute to capabilities and even give a decisive lead. Do you agree with that?
1:10:57
What's your reaction to that concept of interpretability research being very important?
1:11:01
Yeah, I mean we care a lot about it here at OpenAI as well.
1:11:03
So um one thing that we care a lot about is interpreting how the model reasons, right?
1:11:10
um because I think um we've had a very kind of specific and strong view on this um in that we don't want to apply optimization pressure to how the model thinks so that it can be faithful in the way it thinks and to expose that to us you know without any kind of incentives to cater to what the user wants right I think it's actually very important to have that unfiltered view um because you know uh often times like if if the model isn't Sure.
1:11:40
You don't want to hide that fact, right?
1:11:41
Just for for it to kind of please the user.
1:11:43
And sometimes it really isn't sure, right?
1:11:45
And and so uh we've really done a lot of work to try to promote this norm of train of thought faithfulness and and interpretability.
1:11:54
Um and I think it it gives you a lot of uh sense into what the model's thinking and you know what are the pitfalls that it can go off into if it's not reasoning correctly.
1:12:03
That's such an important point because if you have somebody on your team and they come to you and they say, "Hey, you know, I think this is the right answer, but we should probably verify it."
1:12:12
It's like, well, it's still valuable. Totally.
1:12:13
Puts you on the right path.
1:12:15
If somebody comes to you 100% confidence, this is this is the truth and it ends up being wrong.
1:12:19
It's like, well, like trust is just destroyed. Totally. Yeah.
1:12:22
Don't you guys feel like, you know, um, safety felt a lot more theoretical a couple years back, right?
1:12:28
But like today, you know, like the things that people were talking about a couple years, like scalable oversight, like really having the model be able to tell you like and convince you that the work it did was right.
1:12:37
It feels so much more relevant right now just because the capabilities are so strong. Yeah.
1:12:41
I mean, just personally, I' I've completely flipped from being like, uh, oh, the safety research is not that valuable because I'm not that worried about getting paper clipped.
1:12:51
just seems like a very low likelihood that that's kind of like the bad ending like immediately and this fume and all this crazy grey goo scenarios were just so abstract and sci-fi.
1:12:59
It just felt like economics will will will fall into place and there will be uh like a like a cold like a nuclear ending which is like we didn't build nuclear plants and we just stopped everything because we humans seem to be good at that.
1:13:12
Uh but now that we're actually seeing things Yeah.
1:13:14
It's crazy how fast it's been, right?
1:13:16
like um I think my my like my personal story is is like you know what what got me into uh AI was Alph Go, right?
1:13:22
Like just watching it get to that level of capability at Go and you were kind of like it was such an optimistic and also kind of a little bit of a sobering message right when you saw Lisa get beat.
1:13:34
Um, and I just remember, you know, like we we saw the coding models, you know, when we first launched like uh I think very OG codecs, you know, um with with GitHub Code, it was maybe like under, you know, a thousand ELO on um on code forces and I still remember the meeting where I walked into where the team showed my score and they're like, hey, was models better than you?
1:13:54
And it's like you come full circle and it's like, wow, like I put decades of my life into this and you know the capabilities are there.
1:14:02
So like if you know I'm kind of at the top of my field in this thing and it's better than me like what can it do really? Yeah. Yeah. That's amazing.
1:14:10
Uh do I have so many more questions uh on Alph Go?
1:14:12
Are there uh are there lessons from scaling how scaling played out there that you can that we can abstract abstract into the rest of AI research.
1:14:22
What I mean is uh as I remember it, the Alph Go training run was not 100K H200s.
1:14:30
Uh but what would happen if we actually did an Alph Go style training run?
1:14:37
I mean, it would be an economic money pit, right?
1:14:41
Like they had no economic value to do.
1:14:42
But let's just say some benevolent trillionaire decides, I'm going to spend a billion dollars on a training run to beat Alph Go and go even bigger.
1:14:51
um is is go at at some point solved?
1:14:54
Would we see kind of diminishing scaling curves? Could we throw extra RL?
1:14:59
Could we could we port back everything that we've doing in just general AGI research and and and just continue fighting it out in the world of Go or does that end and does that teach us anything? Yeah. Yeah.
1:15:10
Honestly, um I feel like if you really are curious about these mysteries, join our team.
1:15:15
That's the thing I want to say.
1:15:15
Oh yeah, I mean um really like kind of the the central problem of today is RL scaling, right?
1:15:22
And um when you look at AlphaGo, right, it's it's a narrow domain, right?
1:15:26
And I think in some sense that limits the amount of compute you can pump into it, but even kind of small toy domains, they can teach you a lot about how you scale RL like what are the axes where it's most productive to to pump scale in.
1:15:39
Um I think a lot of scaling research just looks like that whether it's on RL or pre-training it's like you identify a lot of you know different different variables under which you can scale and like where is kind of where you get the best kind of like marginal impact for for being scaled there.
1:15:53
Um I think that's a very open question for our right now.
1:15:57
Um and I think what you mentioned as well it's just like you know going from narrow to broad right um does that give you a lever to pump a lot more scale in as well?
1:16:05
Um, I think when you look at our reasoning models today, they're a lot more broad-based than uh, you know, just being able to kind of an expert system on go.
1:16:13
Um, so yeah, I really do think that um, there are so many levers to to scale and what about move 37?
1:16:22
That was such an iconic moment in that AlphaGo Lisa doll match. Uh, they place move 37.
1:16:28
It's very unconventional.
1:16:30
Everyone thinks it's a blunder. It turns out not to be.
1:16:31
It turns out to be critical.
1:16:33
It turns it turns out to be innovation.
1:16:35
Uh do you think we are we're certainly post touring test in language models?
1:16:39
We're probably post touring test in image generation.
1:16:44
Um, but it feels like we're pre-move 37 in text generation in the sense that there hasn't been uh like a fully AI generated book that everyone is just, oh, it's the new Harry Potter. Everyone has to read it.
1:16:58
It's amazing and it's fully a and it's fully generated or or this image.
1:17:01
The images, they do go viral, but they go viral because they're AI.
1:17:06
Move 37 in the context of Go did not go viral because it was AI.
1:17:10
It felt like it was actual innovation.
1:17:12
So, uh, is that the right frame?
1:17:14
Does that make any sense? Yeah.
1:17:16
Um I think it's not the wrong frame.
1:17:16
So I I think some some quick thoughts on on that.
1:17:21
Um I I think kind of um when you have something that's you know very measurable like win or lose, right?
1:17:28
Something like uh like go.
1:17:28
Um yeah, it's like very easy for us to kind of just judge, right?
1:17:33
Like did did the model do something right here?
1:17:34
Um and I think the more fuzzy you get um you know it is just harder, right?
1:17:40
like um when it comes to is this the next Harry Potter, right?
1:17:44
Like you know, it's not a universally loved book.
1:17:45
I think fairly universal, but you know, there's there's some haters.
1:17:48
Um and yeah, I I I think it it is just kind of hard when it comes to these human subjective things, right?
1:17:56
Where um it's really hard to put down in words like what makes you like Harry Potter, right?
1:18:02
Um, and so, um, I think those are always going to lag a little bit, but, you know, I think, you know, we're we're developing more and more techniques to attack kind of these more open-ended, um, uh, domains.
1:18:12
And I don't know, I I wouldn't say that we're not at an innovative stage today.
1:18:18
So, um, I think my biggest touch with this was when we had the models compete on the IY last year.
1:18:26
So highlight it's like the the international basically Olympics for for computer science um basically the the top four kids from from each country go and compete and these are really really tough problems um basically selected so that they require some innovative insight to solve right um I think um and we did see the model come up with solutions even to some very ad hoc problems and and So I think there was a lot of surprise for me there, right?
1:18:58
Um I was completely off base about which problems the model would be able to solve the most, right?
1:19:06
Um I think like I I kind of categorized there there's six problems some of them as more kind of like oh this is standard a little bit more standard.
1:19:12
This is a little bit more out of the box and it's like it's not going to be able to solve this uh more out of the box one.
1:19:16
But it it did and I think um I think that really does speak to kind of uh these models have the capacity to do so especially trained with our own.
1:19:25
Now now put that in context of what's going on with ARC AGI.
1:19:28
Obviously OpenAI has made incredible progress there but it just when I do the problems it seems easy and when I look at the IOI sample problems I think this would be a 20-year process for me to figure out how to achieve that and I can do the RKGI on my phone.
1:19:44
uh is this the spiky intelligence concept?
1:19:48
Is this something that a small tweak in in algorithmic design just oneshots AGI or ARC AGI or or is there something else going on there that we should be aware of?
1:19:59
Yeah, I mean I think um part of this is the beauty of ARGI as well, right?
1:20:04
Like um I think I'm not sure if there's another kind of like human intuitive simpler benchmark which is for the models.
1:20:11
Um and I think really that's one of the things they really optimize for on that benchmark.
1:20:15
Um I do think when it comes to models though like there's just a little bit of a perception gap as well like you know u models aren't used to this kind of native um you know like just screen type input.
1:20:26
Um I think there's a lot we can bridge there actually.
1:20:30
actually. um even O4 mini um it's a state-of-the-art multimodal model in many ways including visual reasoning and I think uh you know you're you're starting to kind of build up the capacity for the models to take images manipulate and and reason about them um
1:20:47
generate new images write code on images and um I think it's just been kind of underfocused but um I think when I talk to researchers in the field they all see this as a part of intelligence too and we're going to continue the focus there. Yeah. Is is is RKGI kind of in the if Yeah.
1:21:01
Is is is RKGI kind of in the if we're dropping a buzz word on it is like program synthesis.
1:21:07
Is there a world where uh I I know that I know the tokens like the images we see them as as renderings of squares and different colors, but uh the when they're fed into the LLM, they're typically uh just a stream of of numbers effectively.
1:21:20
Is there a world where actually adding a screenshot is what's important like visual reasoning? Yeah. Yeah.
1:21:27
So I think I think that could be important.
1:21:29
I think that could be important. It's just like kind of uh you know whenever it comes to like textual representation of grids um models today just don't really do that well right and I think it's just kind of because humans don't
1:21:43
really ever write down textural representations of grids like you know we have a chessboard like no one really kind of just like types it out in a grid like um and um and so the models are kind of like undertrained a little bit on on what that looks like and what that means. Sure. So um you know I I I think Sure.
1:22:00
So um you know I I I think with more reasoning it we we'll just bridge the gap.
1:22:06
Um I think with better visual perception we'll just bridge that gap. Yeah.
1:22:09
How are you thinking about the role of non-lab researchers in the ecosystem today?
1:22:15
I'm sure you try to recruit some of the best ones but the ones that don't join your team.
1:22:19
Tell us about the one that got away. Yeah. The one that got away. Yeah.
1:22:23
No, I mean um I think it's still actually a fairly good time.
1:22:28
time. I for for specific domains right uh to to be doing research and um you know I think the style is just very different um and you do feel the pull of non-lab researchers into labs because I think they feel like a lot of the burning problems in the field are at
1:22:44
scale right um and that's kind of one of the unfortunate things too right like when you look at reasoning um you just don't see that happen at small scale right there's like a certain scale at which it starts becoming signal bearing And that requires you to have resources, right? Um but I do think, you know, a
1:23:01
Um but I do think, you know, a lot of the really good work that I've seen, you know, there's um experimental architectures.
1:23:08
I think a lot of good work is happening in the academic world there.
1:23:12
Like a lot of study in optimization, um a lot of study in kind of like GANs, you know, um there's certain fields where you see a lot of fruitful research that that happens in academia.
1:23:22
Yeah, that makes a lot of sense.
1:23:24
How about uh consumer agents?
1:23:24
How are you thinking about them?
1:23:26
Uh you talked earlier about sort of B2B adoption and that's all very exciting but how much do you and the research or think about breakout consumer agent products?
1:23:39
Yeah, that's a fantastic question.
1:23:41
I think um we think about it a lot.
1:23:43
I think uh that that's the short answer.
1:23:45
Um you know we really do think like this year we're trying to focus on how we can move to the agentic world, right?
1:23:51
And um when I when I think about consumer agents, I think like ChachiPD proved that you know people got it right like people get conversational agents when it uh conversational kind of models but when when it comes to consumer agents we have a couple thesis and um that we've tried out in the world.
1:24:08
I think one one is deep research, right?
1:24:11
Um I think this is something that can do 5 to 30 minutes of of work autonomously, come back to you and really like um kind of synthesizes information, right?
1:24:19
Um it goes out there um gathers, collects, and kind of, you know, compresses the information in in a form that that's useful.
1:24:27
A little bit of a little bit of push back there.
1:24:29
Like I can see that as a consumer product when someone like Aiden is like, I want new towels.
1:24:33
and he uses deep research to like figure out like what is the best towel across every dimension.
1:24:39
But when I think of deep research, yes, it has applications with students, but it's often some of them might just be consumers being like, give me a deep research report on this country and where to travel and things.
1:24:53
We keep using this flight example, but I don't I haven't actually tried to book a flight with deep research.
1:24:57
It's totally possible that it could go and pull all the different flight routes and and calculate all the different delays and all the different all the different parameters of if I fly to this airport, I can park or I can use valet here or something like that. Yeah. Yeah.
1:25:10
And I guess like when I think of agents, it's it's deep research is like, you know, curating information on which you can take action on, but it's like at what point is action a part of that sort of loop, right?
1:25:22
Where you can not only curate a list of flights that you want, but then you know actually go out and and and have agency. Yeah.
1:25:29
I think one of our explorations in that space is operator, right?
1:25:34
operator, right? It's where you kind of just feed in raw pixels from your your your laptop um into or you know from some virtual machine into the model and it it produces you know either a click or some keyboard actions right and um so there it's taking action um and I think the trouble is you know it you don't
1:25:53
ever want to mess up when you're taking action right I think the cost of that is super high um uh you you only have to get it wrong once to lose trust in in a user um And so we want to make sure that that feels super robust before we get to the point where we're like, "Hey, look, here's a tool." Um I that's so different
1:26:10
Um I that's so different than deep research because like you can wind up on some news article and read a one sentence that gets a fact wrong or the commas in the wrong place and the numbers off and but that's just the expectation for just text and analysis.
1:26:28
And if you delegated that, yeah, you're going to expect a few errors here and there.
1:26:31
Oh, that's actually a different company name or that's a that's an old data point. There's new data.
1:26:35
Uh but very different if I book a flight and you book the wrong flight and I can wind up in Chicago instead of New York. Exactly.
1:26:42
And I think the reason why we care so much about reasoning is because I think that's the path that we get reliable agents through. Sure. Right.
1:26:48
Um you know, we've talked about like reasoning helping safety, but reasoning is also helping reliability. Right.
1:26:53
It's like you imagine like what makes a model so good at a math problem?
1:26:59
It's like it's banging its head against it.
1:27:00
It's trying a different approach and then it's like adapting based on what what it failed at last time.
1:27:06
And I think that's the same kind of behavior you want your your agents to have.
1:27:09
It's like things like it adapts and and keeps going until it it's success.
1:27:13
And that that's the humans do this every day.
1:27:15
You're booking a flight, you keep hitting an error.
1:27:19
It's you're not which sure which form you missed, right?
1:27:20
And you're just sort of banging your head against the computer and eventually it says, "Okay, you're booked." Right.
1:27:25
So I think I think that's a great call out. Yeah.
1:27:26
Um, I mean, the there's so many more questions we could go go into, but um, I'm I'm interested in the scaling of RL and kind of the balancing act between pre-training RL and inference, just the amount of energy that goes into getting a result when you distribute it over the entire user base. How is that changing?
1:27:50
And I guess um, is is are we post like really big really big runs?
1:27:54
Is this going to be something that's like continually happening online or it feels like we're moving away from the era of like oh some big development some big run happened and now we're reaping the fruit fruits of it versus a more iterative process.
1:28:10
Um yeah I mean I don't see why it has to be so right.
1:28:13
I think like if you find the right levers you can really pump a lot of compute into RL as well as pre-training.
1:28:19
Um I think it is a delicate balance though between all of these different parts of the machine.
1:28:25
Um and you know when when I look at my role with Yakab it's just kind of like figure out where um how how this balance should be allocated um where the promising kind of like nuggets are arising from and and resourcing those.
1:28:37
Um yeah it's it's kind of a in some sense my I feel like part of my job as a portfolio manager.
1:28:42
Yeah, that's a lot of fun.
1:28:45
Well, thank you so much for joining.
1:28:47
This was a fantastic conversation.
1:28:48
We'd love to have you back and go deeper. Great hanging Mark. We'll talk to you soon. Absolutely. Yeah. Peace. Have a good one.
1:28:54
Uh, next up we have Shalto Douglas from Anthropic coming on this show.
1:28:57
So I I just giving us the update on just getting a lot of messages saying why no one cares about AI.
1:29:04
Talk about the drama on the timeline.
1:29:10
Well, we do care about AI. We care a lot about AI.
1:29:14
Um, but it is a mess out there. Wow. Yeah.
1:29:17
The end of the Trump Elon era. Uh, I don't know.
1:29:20
Well, maybe maybe we have to get some people on to talk about it uh tomorrow or something. Got to do it today.
1:29:26
Anyway, uh we have Shalto from Anthropic in the studio. How you doing? What's going on? Good to see you guys.
1:29:33
Hopefully uh you're you're staying out of the chaos on the Don't Don't open anytime.
1:29:38
Don't open the sweet child. Just Twitter. Mute everything.
1:29:42
Stay focused on the application.
1:29:45
Stay focused on the mission.
1:29:46
Stay focused on the next training run.
1:29:48
We really humanity really cannot afford for any researchers to open X today. What a hilarious day.
1:29:55
Anyway, I mean by 24 hours guys. Yeah. How are you doing?
1:29:58
What what is new in your world?
1:30:00
What what are you focused on mostly daytoday?
1:30:02
And uh maybe maybe as just a way of an intro. Yeah.
1:30:04
Um so at the moment focused really hard on scaling RL.
1:30:08
Um I mean that is the theme of what's happening this year.
1:30:10
Um and we're still seeing these huge gains where you go you know 10x compute increase in RL.
1:30:15
we're still getting like very distinct linear gains on the basis of that.
1:30:19
Um, and because RL wasn't really scaled anywhere close to how much pre-training was scaled at the end at the end of last year. Yeah.
1:30:24
We have like a basically a gamut of re like riches over the course of this year.
1:30:28
So, where are we in that in that RL scaling story because I I I remember the the some of the rough numbers around like GPT2, GPT3, we were getting up into like it cost a hundred million, it's going to cost a billion dollars.
1:30:43
cost a billion dollars. like it just rough order of magnitude not even from enthropic just generally like what is a big RL run cost or or how many are we talking 10k H200s or 100k like are we going to throw the same resources at it and if so how soon yeah so I think in
1:31:00
Darra's essay at the beginning of the year he said that a lot of runs were only like a million dollars back in like December um thinking of like Deepseek V3 and this kind of stuff like R1 um which means that with that's like at least two just to get to the scale of GPD4 and GP4 was 2 years ago. Yeah. Right. Um RL is Yeah. Right.
1:31:13
Um RL is also perhaps a bit more naively paralyzable and scalable than pre-training.
1:31:19
You pre-training you need everything in one big data center ideally or you need like some clever tricks.
1:31:24
Um RL you could like in theory like what the prime intellect folks are doing scale it all over the world out of it and and so we you're held back like maybe like you're held back far less than your train. Sure.
1:31:34
So everyone and their mother has a billion dollars now.
1:31:40
Uh there are there, you know, hundreds of thousands of GPUs getting pumped all over the place.
1:31:44
I I I feel like we're not GPU poor as a as a as a society.
1:31:46
Uh maybe some companies need to justify it in different ways, but it sounds like there's some sort of uh uh like reward hacking problem that we're working through in terms of scaling RL.
1:31:58
What are all of the problems that we're working through to actually go deploy the capital cannon at this problem? Yes.
1:32:05
So I mean think about what you're asking the model to do in RL is you're asking it to achieve some goal at at any cost basically. Yeah.
1:32:14
Um and this comes with a whole host of like uh behaviors which you may not intend.
1:32:18
Um in software engineering this is really easy like to it might try and hack unit tests or whatever.
1:32:23
um in much more longer horizon real world tasks, you might ask it to say go make money on the internet and it might come up with all kinds of fun and interesting ways to do that unless you find ways to guide it into following the like principles that you want it to to obey basically um or to to align it with your like idea of what's sort of best for humanity.
1:32:42
Um and so it's actually it's a pretty intensive process.
1:32:44
It's a lot of work to find down and hunt down all the ways these models are uh hacking through the rewards and and and patch all of that and Yeah. Yeah.
1:32:52
H how uh are we going to see scaling in the number of rewards that we're rlinging against? If that makes sense.
1:33:02
I would imagine that uh at a certain point we unless we come up with like kind of like the the the genesis prompt go forth and be fruitful or something and and multiply uh the you could imagine uh training runs on just knocking down one one problem after another and is that is that kind of the path that we're going down? I I very much think so.
1:33:25
Um there's this idea in which like you know the uh the sort of world becomes an RL environment machine in some respect.
1:33:30
Um because there's just so much leverage to making these models better and better at all the things we care about.
1:33:35
Uh and so uh I think we're going to be training on on just everything in the world. Got it.
1:33:39
Um and then and then does that lead to um more model fragmentation models that are good at programming versus writing versus poetry versus image generation or or or does this all feed back into one model?
1:33:55
Does the idea of the consumer needing to pick a model disappear?
1:33:57
Are we in a temporary period for that paradigm?
1:34:01
I think the main reason that we've seen that so far is uh because people are trying to make the best of the capital.
1:34:10
Like we are all still GPU poor in many ways and people are focusing those GPUs on the sort of like spectrum of awards that they think is most important.
1:34:16
Um and look, I'm I'm a bit of a big model guy.
1:34:21
Um I I really do think that similar to how we saw with large pre-trained models before where small fine tuned models made it like had gains over the sort of GP2 GPT2 era but then were obsoleted by GP4 being generally good at everything.
1:34:35
I think to be honest you're going to see this generalization and learning across all kinds of things that means you benefit from having large single models rather than specialization or area fine-tuned models.
1:34:46
Can you talk a little bit about the transition from or any any differences between RLHF and just other RL paradigms? Yes.
1:34:54
So RLHF uh you're trying to maximize a pretty like lossy signal things like pairwise like what do humans prefer?
1:35:03
And I don't know if you've ever tried to do this like judge two language model responses.
1:35:06
I get prompted for that all the time. Right.
1:35:10
And I'm always like I don't want to read both of those.
1:35:11
I'll just click the one on the left. Exactly. Exactly. Exactly.
1:35:15
You know, I click one of the random ones sometimes. Yeah.
1:35:16
Or or I click like the one that just looks bigger or I'll read the first two sentences, but yeah, I'm not giving straight I'm not I'm not being I'm not doing my job as a as a human reinforcer. Exactly.
1:35:25
Human preferences are easy to hack. Yeah, totally.
1:35:29
Environments in the world are much truer if you can find them.
1:35:32
Um so something like did you get your math question right is a very real and like true reward. Does the code compile? Right. Does the code compile? Exactly.
1:35:41
um you know, did you make a scientific discovery?
1:35:43
We're going to start we we got very little rewards right now, but pretty quickly over the next year or two, you're going to start to see much more meaningful and and long horizon rewards.
1:35:52
You're going to see models bribing the Nobel Committee to win the Nobel Prize. Good reward hacking. Reward hacking.
1:35:59
Thing you want to prevent, right? Exactly. It's Yeah. Yeah.
1:36:01
That's a real nightmare scenario.
1:36:04
Um, what about like there's so many different problems that we run into that feel like the it's just really really hard to divi design any type of eval.
1:36:14
Uh the the uh my kind of benchmark that I use whenever a new model drops is just tell me a joke. They're always bad.
1:36:22
And or or even even the latest VO3 video that went viral was somebody said like uh standup comedy joke and it was kind of a funny joke, but it was literally the top result for joke Reddit on Google and then it clearly just took that joke and then instantiated in a video that looked amazing.
1:36:43
Um but it wasn't original in any way.
1:36:47
And so uh we were joking about like the RLHF loop for that is like you have an endless cycle of comedians running AI generated materials and then and then you know uh speak microphones in all the comedy clubs to feed back what's getting the laughs.
1:37:02
But I mean honestly that would work pretty well actually.
1:37:08
Want to hook us up with an RL loop? I mean Yeah. Yeah.
1:37:09
But but I mean for for some of those less uh like as you go down the curve it feels like each one gets harder and harder uh to actually tighten the loop.
1:37:17
We see this with like longevity research where it's like okay it takes a hundred years to know if you extended a human life like the yes you could create a feedback loop around that but every change is going to be hundreds of years and so even if you're on the cycle it's irrelevant for us in the context that we talk about AI.
1:37:33
So uh talk to me about like are you running into those problems or or or will there be like another approach that kind of works around those.
1:37:42
works around those. So there are a lot of situations where you can get around this by just running much faster than real time like let's say the process of building a like a giant app like building Twitter right is something that would take human months but if you got fast enough and good enough AIS you could do that in several hours right heaps of AI agents they're all building
1:37:58
right you know things spec and so you can get a faster reward signal in that way um in domains that are less well specified like humor I agree it's really really hard and this is like why I think in some respects like creativity is uh like at the at the top end of the spectrum like true creativity much harder to replicate than the sort of like scientific style and that will just take more time. You know what the models actually are
1:38:20
You know what the models actually are pretty good at making jokes about being an AI.
1:38:23
This feels weirdly fresh.
1:38:23
Um like everything else is kind of a weird copy of something like it like it just it feels like it's derivative.
1:38:30
Basically, it's trying to infer what humor is and it doesn't really understand it.
1:38:33
But jokes about being an AI are quite funny. Yeah.
1:38:37
I I I I think this also might be I don't know if it was directly reward hacking, but I noticed that uh one of the new models dropped and a bunch of people were posting these like 4chan like be memes and and and they were it seemed like they were kind of hacking the humor by being hyper specific about an individual that they could find information on online.
1:38:55
And so you're laughing at the fact that it's like, oh wow, that is like something that I've posted about and it's making a reference, but it's not really that funny to me.
1:39:03
It's other than it's just like, wow, they really did its research.
1:39:06
like it really knows Tyler Cowen intimately which is cool but I didn't find it hilarious.
1:39:10
Um yeah yeah very interesting.
1:39:13
Um let let's talk about um some sort of deep research product uh pro projects and products.
1:39:20
Um we were talking to Will Brown and he was saying like AGI is here with some of the bigger models but the but the time that AGI can feel consistent it diverges.
1:39:31
And so you could be working with someone who's, you know, 100 IQ, but they but they will stay consistent for years as an employee or they'll they'll keep, you know, living their life.
1:39:43
Whereas a lot of these super smart models are working really well and then after a few uh a few minutes of work, the the the agents kind of diverge and kind of go into odd paradigms and it feels very not human.
1:39:55
It feels like a like just a they're hyper intelligent in one way and then extremely stupid in others.
1:39:59
Um what's going on there?
1:40:01
uh what is the pro what is the path to extending that?
1:40:04
Is that more like having more better planning and better uh better like dividing up the task or or will this just kind of naturally happen through the RL and scale? Yeah.
1:40:15
So there's that jaggedness, right, which is what you're seeing is how we call it.
1:40:18
And I think that is largely a consequence of the fact that maybe something like deep research is probably being RL to be really good at producing a report. Yeah.
1:40:25
Yeah, but it's never been rled on the like act of producing valuable information for a company over a week or a month or like making sure the stock price goes up in like you know a quarter or something like this, right?
1:40:37
Like it it doesn't have any conception of how that feeds into the broader story at play.
1:40:40
It can kind of infer it because it's got a bit of world knowledge from the you know the base model and this kind of stuff but it's never actually been trained to do that in the same way humans have.
1:40:47
Um so to extend that you need to put them in much longer running much like like you know long horizon things.
1:40:54
Um and so so deep research needs to become you know like deep operate a company for a week kind of thing. Sure. Is that the right path?
1:41:02
Like it feels like the road might be there's a like the longest running LLM query used to be just like a few seconds maybe a few minutes.
1:41:13
And I remember when uh when some of the reasoning models came out people were almost trying to like stunt on it by saying like oh I asked it a hard question it thought for five minutes.
1:41:22
Now deep research is doing 20 minutes pretty much every time.
1:41:24
Um is the path two hours two days or are we going to see more uh efficiency gains such that we just get the 20-minut model the 20-minute results in 2 minutes and then 2 seconds. Yeah.
1:41:36
So this is somewhere where like inference in many respects and and priorization becomes really important.
1:41:40
So both like how fast is your inference literally affects the speed at which you can think and the speed at which you can like like you know do the these experiments.
1:41:47
Also, how easily you can paralyze becomes really important like can you dispatch a team of uh of sub agents to go and do deep research and like compile like sub reports for you so you can do everything in parallel.
1:41:58
in parallel. these kinds of like uh it's it's both like there's an infrastructure question here um that feeds up from the hardware and the chips and this kind of stuff uh to like designing better chips for you know better inference and this all this um and and an RL question of like you know how well can you paralyze and and all this so I think we just need
1:42:17
to compress the timelines compress the time compress the time frames basically so uh if I'm if I'm like an extremely big model and I'm running an agentic process like how how much my hankering for like a middlesized model on a chip or like baked down into silicon that just runs super fast because it feels like that's probably coming. We saw that
1:42:38
We saw that with the Bitcoin pro progression from CPU to GPU to FPGA to ASIC.
1:42:42
Do do you think we're we're at a good enough point where we can even be discussing that?
1:42:50
Because I every time I see like the latest midJourney, I'm like this is good enough.
1:42:53
I just want it in two seconds instead of 20.
1:42:55
Um, but then a new model comes out.
1:42:57
I'm like, I'm glad I didn't get stuck on nothing, right?
1:42:58
But, but yeah, like are how far away from how far away are we from, okay, it's actually good enough to bake down into silicon?
1:43:09
Well, there's a question here of baking it down to silicon versus designing a chip which is like very suited for the architecture that you care about, right?
1:43:16
Um, and baking out of silicon, unsure.
1:43:18
Like, I think that's a bet you could take, but it's a it's a risky one because the pace of progress is just so fast nowadays.
1:43:22
Um, and I I really only expect it to to accelerate.
1:43:24
Um, but designing things that make a lot of sense for the sort of trans, you know, transformers or or architectures of the future should should make a lot of sense. That's a big gap though.
1:43:36
Transformers or architectures of the future.
1:43:38
If we diverge, there's a lot of companies that are banking on the transformer sticking around.
1:43:41
Uh, what is your view on transformer architecture sticking around for the next couple years?
1:43:46
Um, I mean, look, they stuck around for five years, so they might stick around for a little while.
1:43:49
stick around for a little while. U but there's there's different you think about architectures in terms of this balance of memory bandwidth and flops right one of the big differences we've seen here is Gemini recently had actually diffusion model that they released I was about to ask the other day right so diffusion is an inherently extremely flops intensive process whereas normal language model decoding
1:44:08
is extremely memory bandwidth intensive you're designing two very different chips depending on which bet you think makes sense yeah and if you think you can make something that does flops like four times faster than diffusion and like four times cheaper than your others could diffusion makes more sense so there's There's this dance basically between uh the chip providers and the and the architecture. Yeah. Both trying Yeah.
1:44:23
Both trying to build for each other but also like build for the next paradigm. Yeah. It's risky.
1:44:29
Do you I I I don't know how much you've how much you've played with uh image generation, but do you have any idea of what's going on with images in chatbt?
1:44:37
It feels like there's some diffusion in there.
1:44:40
There's some tokenization, maybe some transformer stuff in there.
1:44:42
It almost feels like the text is so good that there's like an extra layer on top almost and that it's it's almost like reinventing Photoshop.
1:44:51
And and I guess the the the broader question is like it feels like an ensemble of models maybe the discussion around around just a agents and textbased LLM interactions shouldn't necessarily be uh transformer versus diffusion, but maybe how will these play together?
1:45:06
Is that a reasonable path to go down?
1:45:08
Well, I think pretty clearly there's some kind of like rich information channel between even if there are multiple models there, there's like it's it's conditioning somehow on the on the other model because we've seen before like let's say when uh you know models use midjourney to produce images, it's never quite perfect.
1:45:25
It can't perfectly replicate what went in as an input.
1:45:26
It can't perfectly like adjust things.
1:45:27
U so there's a link somehow whether that's the same model producing tokens plus diffusion. I don't know.
1:45:34
Um like yeah, can't comment on what open is doing there. Yeah. Yeah. Yeah.
1:45:38
Um are are there any other kind of like super wildcard longshot uh research efforts that are maybe happening even out even in academia where I mean this was the big thing with uh what was his name Gary who's talking about I forget what it was called symbolic symbol manipulation was a big one and and I feel like you know you can never count anyone out because it might come from behind and be relevant in some ways.
1:46:06
Um but but are there any other research areas that you think are like purely in the theory domain right now that are worth looking into or tracking that you know low low probability but high upside if they work? This is a tough one. This is a tough one.
1:46:25
But I will say it's not a symbolic thing.
1:46:26
It's crazy how similar transformers are to systems that manipulate symbols. Sure.
1:46:31
Like what they're doing is they're taking a symbol and they're like converting it into a vector and then they're manipulating and moving like information around across them. Sure.
1:46:37
Like this this whole like uh debate that all transforms cannot represent symbols and they cannot do this.
1:46:44
I think it's it's yeah it's not real.
1:46:46
So so Gary Mark is underrated or overrated I guess. Overrated. Yeah. Yeah.
1:46:51
But uh but I mean if I if you if you twist the uh if you twist it so much you wind up with saying like well really like the the transformer fits within that paradigm and so maybe it's you know it it's you know it like the rhetoric around it being a different path was maybe false the whole time. Something like that.
1:47:11
the whole time. Something like that. Um but but I as I remember that debate, it was really the the idea of compute scaling versus almost like feature engineering scaling and and will the progress scale with human hours or GPUs essentially and that has a very different economic equation and it and
1:47:33
it feels like there's been there's been some rumblings about maybe with a data wall we'll shift back to being human labor bound, But do you think that there's any chance that that's relevant in the future or is it just algorithmic progress married with bigger and bigger data centers in the future? So I'm
1:47:50
So I'm pretty bitter lesson built uh in sense that I do think removing as many of our biases and our like clever ideas from the models is really important just like freeing them up to learn.
1:48:02
Now obviously there's like there is clever structure that we put into these models such that they're able to learn in this extremely general way and that uh but I I am more convinced that we will be computebound than than we will be like human researcher uh human researcher bound on on this kind of thing like we're not going to be feature engineering and this kind of stuff.
1:48:23
We're going to be trying to devise incredibly flexible learning systems. Yeah, that makes sense.
1:48:27
Um, on the scaling topic, part of I I I part of my like worry is that the the oos get so big that they turn into these mega projects that are uh that at a certain point you're bound by the laws of physics because you have to move the sand into silicon chips and you have to dig up the silicon and at a certain Yeah.
1:48:49
there's only so much sand and like the math gets really really crazy just for the amount of energy required to to move everything around to make the the big thing.
1:48:59
Uh where are you on on how much scale we need to reach AGI?
1:49:05
How whether or not we will see um like the laws of physics start acting as a drag on progress because uh it certainly feels exponential.
1:49:14
We're feeling the exponentials but uh a lot of these turn into sigmoids, right? Yeah.
1:49:20
Um, so I think we've got what, like two or three more before uh before it gets really hard.
1:49:25
Leopold has this nice table at the end of his situational awareness where I think like 2028 or something is when uh under really aggressive timelines that you get to 20% of US energy production.
1:49:35
It's pretty hard to go exponentially beyond 20% of US energy production.
1:49:39
Um, now I think that's enough.
1:49:42
Every indication I'm seeing says that's enough.
1:49:45
Now there there might be some complex uh you know data engineering, rule engineering, this kind of stuff that goes into lots there's still a lot of algorithmic progress left to go.
1:49:54
Uh but I think that with those extra oos we get to basically uh a model that is capable of assisting us in doing research and software engineering. Yeah.
1:50:06
Which is the beginning of the selffor. Yeah. Interesting.
1:50:09
Is that just a coincidence?
1:50:10
Like this feels like one of those things.
1:50:12
This feels like one of those things where like the moon is the exact same size as the sun and the sky.
1:50:16
It's like, oh, it just happens that AGI happens within this time.
1:50:19
Like, did you have you unpack that anymore?
1:50:21
Because it feels convenient.
1:50:23
Not not to, you know, I know less.
1:50:25
I mean, there's there's a lot of weird conveniences or like weird.
1:50:29
It's a good sci-fi story, let's say. Totally.
1:50:30
You know, we've got, you know, Taiwan in between China and the US and it produces the most valuable material in the world and that's locked between the two. Incredible plot. Incredible plot. Yeah.
1:50:40
really bad for the people that don't think of that don't believe in simulation theory.
1:50:43
It really feels like all this is scripted. It's fascinating.
1:50:47
Um talk to me more about um uh getting to an ML engineer in AI and and kind of that reinforcement.
1:50:55
AI and and kind of that reinforcement. I imagine that you're using AI codegen tools today and and anthropic is broadly and everyone is um but but uh what are you looking for and what are the what's the shape of the the spiky intelligence where do they fall flat and what are you looking to kind of knock down in the
1:51:14
interim before you get something that's just like go yeah so I mean we definitely use them the other night I like I was a bit tired I asked her to do something just sat watching it in front of me working for half an hour it was great it was truly weird experience particularly when you look back a year ago and we're still copy pasting stuff between a chat window and and you know a code file. Um uh what I like meters
1:51:32
Um uh what I like meters evals for this kind of stuff.
1:51:36
So they have a bunch of evals where they measure like the ability to write a kernel, the ability to run a small experiment and improve a loss.
1:51:43
Um and they have these nice progress curves versus humans.
1:51:45
Uh and I think this is maybe the most accurate reflection of like what it will take for it to really help us um at doing progress.
1:51:53
And there's a mix here like where they're not so great at the moment is like large scale distributed systems engineering, right?
1:51:58
Like debugging stuff across heaps and heaps of accelerators and like the where the feedback loops are slow and you actually like the if your feedback loop is like an hour, then it's worth you spending the time on on doing something.
1:52:08
If your feedback loop is 15 minutes if it's and and for context there the hourlong feedback loop is just because you have to to actually compile and run the code across everything that spin up all your machines or you need to like you need to like run it for a while to see if something's going to happen like at that point in time you're still cheaper than the chips and so uh you're you're you're sort of it's better that you do it.
1:52:31
sort of it's better that you do it. Um but for things like you know kernel engineering uh or for uh like you know actually even just understanding these systems incredibly helpful like I one thing I regularly do at the moment is in parts of the codebase in like languages that I'm unfamiliar with or stuff like this I'll just ask it to rewrite the entire file but with comments on every
1:52:49
line game changing it's like comments yeah or just hunt through like thousands of files and explain how everything interacts to me draw diagrams this kind of stuff it's really yeah yeah how important is a bigger context window uh in at in that example you gave that feels like something that's important and yet it I I I just naively like Google's the one that has the million token context window. I imagine that all
1:53:10
I imagine that all the other frontier labs could catch up but it seems like it hasn't been as much of a priority as maybe like the PR around it sounds like is that important?
1:53:20
Should we be go should we be driving that up to like a trillion token window?
1:53:24
Um is that is that just going to happen naturally?
1:53:25
There's a nice plot in the Gemini 1.
1:53:28
5 paper uh where they show the like loss over tokens as a function of context length and they show that the loss goes down quite steeply actually as you put more and more and more like of a code base into context you get better and better and better at predicting the rest. Yeah, that makes sense.
1:53:39
With context length, it's a cost.
1:53:40
Um you know the way transformers work is that uh there's you know you have like this this memory that is proportional the KB cache is proportional to how much context you've got.
1:53:52
Uh, and so you can only fit so many of those into like your various chips and this kind of stuff.
1:53:56
Uh, and so a longer context actually just costs more cuz you're taking up more of the chip and you're sort of like you could have otherwise been doing other requests basically.
1:54:04
So bringing it back to the custom silicon, is that a unique advantage of the TPU?
1:54:07
Is is that something that Google has has thought about and then wound up to put themselves in this advantage position or is it a durable advantage even? Yeah.
1:54:15
So GPUs are good in many respects partially because you can connect hundreds or thousands of them really easily across really great networking.
1:54:23
Um whereas only recently has that been true for uh GPUs like GPUs. Yeah.
1:54:27
With MVLink and like the MVL 72 stuff. Okay.
1:54:30
Um so it used to be like eight GPUs in a pod and then uh like you connect them over worse uh interconnect and now you can do 72 um and and then it breaks down.
1:54:38
With Google Tus you can do like 4,000 8,000 over really high bandwidth interconnect um in one pod.
1:54:45
Um and so that that is helpful for things like just just general scaling in many respects.
1:54:48
Um I think it's this is doable across any chip platform but uh it is a is an example of like somewhere that being fully vertically integrated is a is isn't a benefit. Yeah, that makes sense.
1:55:00
Uh talk to me about arc AGI. Why is it so hard? It seems so easy.
1:55:04
It does seem easy, doesn't it?
1:55:07
Uh that's a well it certainly seems like more more evaluatable than tell me a funny joke, right? Yeah. Yeah.
1:55:13
And I I mean I think if you rled on RKGI then it would you'd probably get superhuman at it pretty fast.
1:55:19
Um but I think we're all trying not to RL on it so that it functions as like an interesting held out. Sure. Okay.
1:55:26
Wait, is that just an informal agreement between all the labs?
1:55:30
Basically, you know, we're trying to have a sense of honor between us. That's honor. Um that's amazing.
1:55:33
How many people on Earth do you think are getting the full potential out of the publicly available models?
1:55:40
because we're now at a point where we have, you know, billion plus people are using AI almost daily and yet I have to imag my sense would be it's maybe like 10,000 20,000 people on the entire planet are getting that sort of full potential, but I'm curious what your assessment would be.
1:55:56
Yeah, I I completely agree.
1:55:58
I mean, I think that even I don't get the full full potential out of these models often.
1:56:01
out of these models often. Um, and I I think as we shift from you're asking your questions and it's giving you sensible answers to you're asking you have to go do things for you um that might take hours at a time and you can really like priorize and spin that we're
1:56:16
going to hit like yet another inflection point where even less people are like really effectively using these things because it's basically going to require you to like it's like Starcraft or Dota like it's going to be like your APM of like managing all these agents and that's going to be quite process. Yeah. Yeah.
1:56:31
Starcraft is such a good example.
1:56:31
You think you're just absolutely crushing it and then you realize like there's an entire area of the map you're just getting destroyed on. Yeah, exactly.
1:56:37
Um it's such a good it's such a good comp. That's great. Anything else, Jordy?
1:56:43
Um I think that's it on my side.
1:56:46
I mean I I would like this to be an evolving conversation. Yeah, this is fantastic.
1:56:51
We'd love to conversely fun back on.
1:56:52
Yeah, we'll talk to you soon. Cheers, Shelton. Have a good one. [Music]