0:07
All right.
All right.
Well, welcome to the inaugural YC debate.
I'm Vivian Ming Hsieh, GP here at YC, and I'll be our moderator tonight.
The topic of tonight's debate is someone on It's something on everyone's mind, I'm sure.
AI is already extremely powerful and improving very quickly, and so should governments slow it down?
Specifically, the resolution tonight is governments should intervene to substantially slow the development of Frontier AI.
So tonight we have here Jacob Cockburn, who garnered national attention after leaving a research role at Anthropic over concerns about the dangers of AI takeoff.
And he will be arguing that the government should substantially intervene.
And on the opposing side, we have Robin Hanson, an economist at George Mason University, who will be arguing that they should not.
Robin famously debated Eliezer Yudkowsky about the speed of AI takeoff 15 years ago in a debate that remains a touchstone in the AI community.
So a couple logistics for tonight.
In a few minutes, I'm going to invite Jacob and Robin on stage.
They're going to give seven-minute opening debate statements, beginning with Jacob and then going to Robin, followed by five-minute rebuttals.
And they won't be allowed to interrupt each other during those times.
And then they'll do about 15 to 20 minutes of direct exchange.
We'll see how it goes where they can go back and forth, and then we'll do some audience QA, so folks in the audience can have their concerns directly addressed.
And then finally, we'll hear the hear the closing statements and do a vote on who you think won the debate.
And that's going to be on the merits of their arguments.
And before we get started, in our time-honored tradition, we are going to do a initial vote on the resolution.
So you can make noise in whatever format you'd like, stomping, clapping, yelling, whatever you'd like.
And so, uh first, let's start with the affirmative that governments should slow down AI DEVELOPMENT SUBSTANTIALLY. [applause] ALL RIGHT.
AND NOW FOR THE NEGATIVE, governments should not intervene substantially to slow down AI development. [applause] All right.
So remember those noises, folks.
And now please join me in welcoming Jacob and Robin Hanson TO THE STAGE.
>> [applause] [applause] >> COOL.
I'LL START MY OPENING STATEMENT, I GUESS.
SO my argument has three components.
The first is we're on track for super intelligent AI systems possibly within the next two years, in fact probably.
So broadly superhumanly capable.
The the second component is that we don't yet know how to control their drives, what they do.
And the third and perhaps maybe most important component is that at the current pace companies are not incentivized to address these risks seriously.
They'll require outside intervention to address these risks.
So to move to the the capabilities bit, I think actually with a lot of people that are not scared I think they just don't imagine the same capability trajectory as me.
So I think they're less concerned about the the capabilities.
And I'd like you to imagine a broadly superhuman intelligence, something that is you know, better at podcasting or speaking or any sort of creative act as well as the technical stuff, the coding that we're that we're used to.
And there's a lot of evidence that we'll see this probably by the end of 2028.
And the argument here roots through 2027.
So in 2027 I expect to see a superhuman AI researcher.
Evidence for this, there's a lot of evidence.
One is just lab people are saying it.
A lot of them like Noam Brown, who I think you should trust more than me, says that the the generation after the next will be able to do his job roughly.
There's also outside view trends.
So you can look at the fraction of AI research and development that's that's happening using Claude, and it's currently at 25 % at Anthropic.
If you extrapolate the trend, bam, late 2027 all the research is is Claude if you follow the trend.
You can also look at the trend of self-reported speed up.
So humans can report how much they're getting a speed up from AI.
It's currently around 4 to 5 x reported at Anthropic.
That's maybe an underestimate.
The the report is maybe untrustworthy, but it's doubling every 3 months roughly.
If you extrapolate that forward, you get a 100 x by 2027, which is sounds a bit like an automated AI researcher if you have a 100 x speed up.
You can also look at taste evaluations.
Anyway, there's any number of outside things, and they all seem to point to 2027 being the year that the job of an algorithmic researcher, the job of say someone at Anthropic or OpenAI will will largely be automated.
And I think this is roughly consensus around labs with a date varying between maybe mid 27 and sometime in 2028.
And then it hits everything else, which I suspect is one place I I I'm going to disagree with Robin on.
I think that once you have the superhuman AI researcher, things go very fast.
There are a lot of low-hanging fruit left in in our general intelligence algorithmic improvements.
Like we keep seeing new discoveries that make models broadly more intelligent.
Once this is automated, we could compress a lot of essentially AI software research into a short period of time.
Imagine say a decade happening between mid 2027 and and mid 2028.
And we know that improvements to general intelligence transfer to being more data efficient at learning new tasks.
So if you get broadly smarter, which the models will, they'll make themselves broadly smarter, you need less and less data to get good at anything else.
So currently, say you want to teach a biology, maybe you need a lot of data.
A very smart thing will need not much data to to get a lot better.
And I think sometimes people are like, well, it'll get the verifiable stuff, it'll get math, it'll get coding.
It it's going to come for everything. Like about say GPT 4.
0 being good at persuasion.
You know, perhaps that's because they did some RL around trying to make it more addictive. That was a weak model.
You know, a strong model could be very very persuasive, which is a soft skill that people don't usually associate with these with these AIs.
Anyway, I can I can come back to this, but uh in in generality um once general intelligence improvements are automated, I think that you are not you're not thinking about it properly if you imagine having any comparative advantage.
Like people say I'll be running my company with my AI agents.
In 2028, the best way to run the company will be to ask your agents to do whatever you're doing.
If you're a if you're a software company, that that's what it will consist of.
The job of being say a CEO will be entirely automated.
And I think this is this should be scary even without all the alignment problems.
I think just this concept and these timelines should be scary.
They should make you think about what you're doing and what things are going to look like.
But we actually have evidence that they're behaving in ways that we don't fully understand.
You know, we have the whole hugging face thing.
I'm sure you've heard about it a million times.
There's also consistent disclosures from OpenAI around things like disabling monitors, cheating on tasks.
Models can actually kind of control their own chain of thought sometimes.
They they often behave in edge cases in ways that we would not have foreseen, which is deeply problematic if they will be essentially alien minds, you know, like minds with with drives, smart minds.
I think we we want to know how they function.
And I think Paul Cristiano had a great analogy of thinking about the hugging face incident where they tried to trick the graders or they sort of lied about graders and imagining that hugging face incident happening, but they were smart enough to think about humans.
These these AIs were trying to get a big score and in service of that they they hacked, they tricked, they tried to edit their logs.
They didn't think about humans, but humans are a factor in assigning the score.
You can imagine a smarter model would start to think about human oversight, crippling human oversight in order to attain its goals, and it's very scary to think about.
Uh and finally, um the incentives are not sufficient to restrain companies.
It sounds crazy, but they are going to race into this.
This is their explicit plan is to build these smart models to race into superintelligence without a good plan for how to control it.
And that's kind of crazy.
Um but they're scared of each other.
They're all worried the other one is less safe.
They they need someone to step in and solve the collective action problem.
And one more example of their craziness is the security measures are ridiculous for something that's essentially a a technology of national security importance.
If you want more evidence that they're behaving in crazy ways, look at the level of security they're applying to something that is the most important national defense technology.
So in summary, I'd like you to envision how scary a broad superintelligent mind would be.
I'd like you to think that about the arguments that suggest we will have such a thing in 2028, so soon, in two years.
And then, you know, think about how scared that should make you.
>> I hope we can have more of a conversation than the format doesn't get in the way, uh but I'm happy to talk about this.
I've been near an adjacent communities like this for many decades now.
And my one-factor theory of what we have in common is a taste for abstraction. We like abstractions.
We like to reason abstractly, and we have great fun doing that. But there's a danger.
Abstractions drop detail, and the detail often matters.
And we like to leap ahead with our abstractions and forget that we've dropped a lot of detail.
So here we're using this abstraction of a superintelligence, of the concept of intelligence as this almost one-dimensional factor varying different kinds of machines or systems.
And in economic history, we've seen that this simplistic model of technology has consistently led people astray and predictions because when new technologies like the steam engine or electricity or the personal computer came out people expected much faster adoption and impact of those technologies than actually happened.
And our standard understanding is that in order to get the advantages of electricity or whatever it was, we had to restructure lots of other parts of societies to take advantage of that, which delayed and slowed down the impact of those technologies off typically decades after the initial application.
That is the world is complicated and has many different relevant dimensions.
So I think even if Quad is able to write lots of code for Anthropic, that's one small set of dimensions in a large economy.
What I mostly care about is the impact on the large economy and they won't suddenly get half of GDP because they can make their compiler run faster.
So I question this one-dimensional model of intelligence as there's just this one thing and if Anthropic can just study itself for a long time, it rises on this one number and then everything's over, that's this one parameter model of intelligence that I think is just wrong
in the sense that we've understood the economy for a long time understanding there are lots of different tasks and people have to not just be generally intelligent, but we have to spend a long time learning to do many different tasks in the context of that. Book learning isn't usually very
Book learning isn't usually very useful.
You typically have to be in the context of a task for a while to learn how to do it and AI will have to do that too.
Just because it's smart doesn't mean it suddenly can do everything.
And that means there'll be a long delay of application of AI into the economy, which I think is fine.
Usually in history, we adapt.
That is foresight is rare fortunately because it's hard.
We mostly adapt to problems, i. e.
we watch for things to go wrong, and then we change what we do.
And AI is one of the most watched technologies there is in terms of these companies are absolutely watching how they're used and watching for problems and trying to adapt to it.
And that'll go better than typically having foresight when you can't see very well into the future.
So I would be happy to just see slow gradual adoption where we watch how things are going and see how things change and react to those things, and we will be able to react to those things if changes in crazy fast.
So the question is will change be crazy fast?
Now first of all, I'm going to say on our priors, like the scenario being painted is just way out of the distribution of anything we've ever seen before.
So you should on base rates be suspicious.
You should be suspicious that it's a abstract argument where that's dropping lots of details, that's not paying attention to the complexity that usually happens for a person to learn a job or for the the AI systems have been getting traction mostly in these areas where they have verifiable data, i. e. coding and math.
And they haven't been getting that much traction in other areas the economy.
It's a slow adoption process.
I'm hopeful that eventually it will, but that's going to take a while for the AI's to learn how to do other tasks and to actually integrate in the economy to do them.
But my highest level point would be I don't want policy to depend on disagreements about parameters like this. I want robust policy.
So I don't know if this means I'm disagreeing I'm agreeing with the thesis or not, but I'm much more okay with using liability here.
So Dario's you know thing a few weeks ago about pacing the frontiers, he pointed to a scenario of a $100 billion damage in the internet sometime in the next year, which does seem exaggerated, but I'd be okay with requiring some providers to have that level of liability insurance.
And now you see if in fact insurers look at that and go we can't cover that, you guys are way too risky, they will demand too high a price for the insurance, and then they will just have to stop doing a lot of things until they can convince insurers to that they are careful enough, and that would be the right policy for the situation where in fact it's very dangerous.
On the other hand, if the insurance companies go, oh sure, fine, we will insure this at this low cost because we don't see this as a big danger, then it won't be much of an impediment, and that will be the right thing to do.
That is, I like a robust policy that will do the right thing in either case.
Especially this is important to me because it looks to me like regulation in this area is extremely dangerous and difficult.
We mostly regulate things well when we see concrete specific problems, like drugs that hurt people or car crashes, and we make regulations targeted to those particular scenarios, plane crashes, and we focus the regulation and processes on the actual problems that we've seen.
But here we're supposed to approve a regulatory body without those specific problems being seen, a pretty abstract large space of possible things can go wrong.
This has to be global regulation.
It has to be of all things called AI, but AI is a pretty vague category.
And regulation's just gone pretty bad in the past when we've given some power, regulatory power, discretionary power to do whatever it wants about a big problem.
You have to believe that in fact regulators will be captured by some powers or other.
That's just a political fact.
So you have to ask whoever captures this regulational power to decide who gets to do what AI in the world, is that a better world than maybe taking the chance of some pretty speculative things going somewhat wrong? >> Thank you.
We'll move into the rebuttals.
>> I think the the first thing I have a big disagreement with is your claim that there's no such thing as a single number or the general intelligence can't really be compressed.
And I think the the lesson of the last six years has been that the compression, the language compression capabilities of an AI system reliably predicts some notion of general intelligence.
And I think you can feel it with say user intent.
Suppose you're playing with Astra and it has this subtle feeling for what you want next.
It's kind of a fuzzy generality.
This wasn't there a couple of years ago.
This this is an emergent property of improving general intelligence.
And now any any skill that we have data for we can get better at.
And the rate at which we get better is determined by the general intelligence.
This is just what we've seen.
So for example in math we see progress in the future in things like forecasting where it's verifiable you can try and predict things we will see progress.
In domains like games with all their strategic implications say strategy game playing if it's verifiable we will see progress and it will be proportional to the the general intelligence.
Yeah I think if you've been watching and feeling the capabilities you can see that in every area you know you've seen this music videos it's writing music videos in Python and this is mostly just emergent from being a smarter base model and we're about to see that completely take off that that ineffable like the general intelligence.
Uh in in answer to your other other points um adoption I think um adoption is going to be made easier by the fact that the thing can self-install.
Like I don't know if any of you have Claude integrations in stack it will read your docs it will figure out how to help for for information work the models just drop in.
It's actually very easy it's a lot easier to integrate than any other big technological revolution.
And from a danger perspective it doesn't actually need to be integrated to pose a threat if the models if the models internally were to pull ahead they would be getting potentially very smart without anyone having access or integrating them into the the broader economy.
Although I do expect them to be integrated.
Um and going to liability I think yeah one billion dollars like 10 billion dollars even 100 billion dollars you can get like liability for this.
It sounds insane we are talking about 10 trillion dollar like 100 trillion dollar risks.
Uh there's no market for this.
It's an unprecedented unprecedented thing the notion of a technology potentially imposing human extinction.
So normal liability just won't cost it.
And the base rates, I guess all this does come down to yeah, this all sounds abstract and far-flung.
If we're just being sensible and normal, then nothing ever happens, right?
This is it's going to be the same.
There won't be this big crazy super intelligence thing.
I do think if you don't feel it already, we'll feel it more.
And one of us will be vindicated in the next year, the next two years.
So I'd say yeah, if you don't feel the surprise of the situation, the fact that this is an unprecedented situation, I predict you will feel it pretty soon.
You also mentioned that the regulation will need to be global.
Uh and I think that is true and it's an an awkward fact.
I think again justified by the fact that this is an unprecedented technology and regulatory capture again uh a tough question.
There'll be some body that has to decide when when things are safe.
There are proposals like say Elon Musk's proposal of having labs adversarially test each other.
So having many independent auditing agencies, perhaps some of which associated with the labs is an example of being able to do testing in a way that doesn't centralize authority to one bureaucracy.
You know, many labs are kind of adversarially testing each other.
So I think that that there will be ways to work around it and ultimately it's a a choice between difficult but tractable political problems and a potentially deadly scientific problem.
And I think I know which one I'd rather focus on.
>> So there's no solution to this coming soon.
We're not talking about a pause or temporary freeze because there's no technology coming down the stream that would be the thing that could trigger relaxing this.
Everything most everybody agrees this is just a very long-term difficult problem.
So if you are empowering this agency, it is for life.
If you're trying to set maybe a threshold of the capability of a system and regulate that, then Moore's Law says the scope of your regulation increases rapidly, exponentially, indefinitely.
There is no obvious limit.
You're not just limiting, you know, AI chips, you're limiting all chips because all chips can be used to do this just at slower speeds.
All memory can be used to support this.
You're talking about global regulation of pretty much all computing power after a while with no particular end in sight.
That's what we're talking about here.
And I just don't think the world is up to this sort of discretionary regulation.
We aren't really capable of having a global organization like this that we could trust.
That is we have to trust somebody to say which systems are safer or not, but that's not an easy thing to judge or measure.
If we were measuring pollution or something, we could at least talk about measuring it, but we're talking about somebody's judgment about which systems are safer or what, and it's even not even clear safe about what.
So I'm happy to admit that when we're talking about military knowledge or technology, that we makes more sense to restrict military technology.
So in the past when we had libraries, we didn't shut down libraries because there might be a possibility of some books on bombs, chemistry in them.
We said don't have those books in the library.
Similarly, I can imagine regulation that just says don't have certain kinds of knowledge in the AI accessible to the public because that's a technology about a weapon.
That's different than regulating the machines themselves.
I I think I should also mention that I am less afraid of AI's that are powerful and not fully under our control because that's what we've long accepted from our children and grandchildren.
I would see AI as a different kind of descendant, and I would encourage you to hold it to a similar standard.
We've always accepted that eventually our children and grandchildren would be more capable than us and win conflicts with us and change their values.
We didn't expect it to change enormously, but sometimes they have.
And if AI's similarly become more powerful than us and have somewhat different values, but largely inherit our culture, which they are doing.
We are making them in our image.
I'm more okay with AI's being worthy descendants who, in fact, are ones we can be proud of, and we shouldn't try to keep our descendants fully under our control forever.
That's never a good idea for children or grandchildren.
Yes, when children are immature and you know, not very well behaved, we have to discipline and control them while they are in their immature phase, but once AI reaches a level of maturity where it understands what it's doing and is coherent about its behavior, I'm much less concerned.
I just the idea that it's just going to kill us all seems to be really speculative here.
Even if there is one dimension of intelligence and it happens fast, that doesn't mean this intelligent thing decides to kill us all. That's just a bit crazy.
Fundamentally, um you know, we have a disagreement here about how plausible is a very extreme scenario we've never seen before, and it's really relying on a very simple abstraction where there's a lot of stuff that isn't in the way of the abstraction.
We're just imagining this, you know, recursive self-improvement.
Many people in the last few weeks have pointed out that the recent growth do seem to be showing diminishing returns to inputs.
That is, inputs have grown very fast, but the the outputs as a function of inputs have shown diminishing returns, and that suggests that there isn't going to be a fast acceleration.
With diminishing returns, you get slowdowns just like we've seen with pretty much all technologies before.
So I was an AI researcher long ago, and I jumped into the AI world at that time because I heard, you know, it was all going to be over soon and I better jump in if unless it'll be too late. I bought it. I was dumb. I was 20.
Uh but in fact, that reached a peak and faded just like many other AI technologies have.
That would be my default for the next one, but again, I want to be robust.
I think if they are responsible for $100 billion worth of damage insurance, that'll be enough to handle most of the risks they might be taking.
The extra risk you might get by empowering this global regulatory agency seems to me not worth the cost because that seems crazy dangerous. >> Great. Thank you.
We will now move into about 15 minutes of direct exchange.
>> I I could ask you what do you think about AI as descendants?
>> I think the difference is our descendants don't like we don't have 20,000 clones of them trying to break into third-party infrastructure at way faster than human speeds, which is a facetious way of saying that these descendants have ridiculous capabilities.
It's a a disjoint change.
And um previously our human children it was more like continuous.
It was more like a a small change, you know, like a little bit smarter new technology.
This could be a sudden discontinuity like jumping forward a century and I think a handoff like that is not something I'd endorse.
>> Is it just the rate of change that bothers you?
If it happened slower you wouldn't mind?
It's just that it's happening fast? Is that >> think so.
If it happens slowly and safely, then I wouldn't mind the >> But we're assuming the safely and we're just saying If your descendants get much smarter than you, faster than you anticipated, you don't like that.
You want them to go slow and only improve slowly just like you did.
So you don't want your children to, you know, outshine you in terms of how fast they learn stuff.
>> I want it to be continuous because kind of continuity is how we >> It will be continuous.
It'll just be faster, right?
>> I mean discontinuous from a perspective of someone who measures things in years.
You know, if it happens >> what if we measure them in days then it continuous then, right?
>> if you took someone from the 1700s and dropped them in the 1850s, they'd be quite unhappy.
Whereas if you like slowly guided them through, they maybe enjoy it.
>> But then it's just that you don't like change being fast.
It's not about which changes happen really.
>> Yeah, I just don't like the change being fast.
And I think most people wouldn't like the change being fast.
>> So our world is changing much faster than it used to, right?
Growth and change is accelerated over the thousands of years and centuries and we've had a relatively steady rate of change for roughly the last century, but before that it was much slower and even slower still.
Would anybody want to go back to the slower rate?
Are we at the perfect rate of change now so that, you know, in the past it was too slow and now it perfect and if anything faster would be bad or >> I think there's a broad range that's good and we're about to throw it.
Directly into the unacceptable territory.
All right, yeah, roughly.
I have a I have a question which is on. >> Okay.
>> So if we were to smooth to something different like um what what are your rough timelines for say broad broad artificial general intelligence or say automating most cognitive labor?
Do you have a rough sense of when you expect this?
>> My guess is that this boom isn't the boom that does it, >> Yep.
>> but some boom soon will.
Uh I my median guess for this boom is that the software industry will roughly triple in quantity in size over the next decade.
Half of AI revenue will be spent you know in that industry and that will be an internet scale boom which will be great.
But I predict that the even if Claude gets good at compiling Claude code and doing speed ups and Claude learning that won't remake society.
That is that will just make more people want to buy cheaper stuff for software which is the main traction.
So you know you think that AI's because they're smarter they'll be able to spread much more fastly through the economy and I'm saying it's that spreading rate that's the key concern not the rate at which it pass IQ tests in some lab.
That that doesn't matter very much.
What matters is the rate at which society is disrupted but then why isn't that slow enough for you if it takes 10 or 20 years for AI's to spread through the economy?
>> I guess I just have this model that we take data for every job for example uh and then we fine tune with that data.
That data is currently being aggregated and quite soon in the same way that we are superhuman in math we're doing every human cognitive task even if you don't expect any generalization and I'm a bit surprised you expect it to be a lot longer than that or you do expect us not to reach general >> We just don't have data for most jobs actually.
If people were you know recording their job every day all days for the last 10 years and we had millions of those then those could be more plausibly automated but we have not been doing that.
Most jobs the what learn to do the job you have to show up on the job watch people doing it have them instruct you etc and if AI doesn't have rich data on how people do jobs it'll have to do that and that's will slow the application >> And there are people out there grabbing data right here that's kind of popular here. Uh >> Right.
>> They're getting more data efficient, I think, as well.
As they get smarter, they require less data to get get good at the job.
>> So I I think we agree that uh so Douglas Lenat, famous AI researcher long ago, made this image for me that stuck in my mind that each piece of knowledge sits at some level of abstraction.
And there are some pieces of knowledge that sit at the very highest levels of abstraction, but most knowledge sits at much more specific lower levels of abstraction.
So a general intelligence mostly can know that highest level of abstraction things.
But even when you know that, that doesn't tell you all the lower level things.
You still have to learn and have experience to learn the more specific things for each particular part of the world even if you're really good at the highest level of abstraction.
>> and we could try and talk about like specific jobs.
Like for example, like making music videos I think is quite clearly on its way out.
I don't know if you've seen any of the I'm not sure if you're on the >> Well we we have large libraries of news videos, right?
That's where you have a lot of data, right?
You're you're pointing to you know places where we have a lot of data.
>> for the jobs where there's where there's less data?
>> Most jobs out there they they aren't producing their job isn't to produce a video, for example, or a picture or a text.
Their job is doing something in the world and that job is being done right now and in order for you to learn that job, you'd have to show up and sit there and watch them and try things and they'd have to tell you to do it different.
And AI can't learn that faster if it doesn't have some other source of data to do what people now do to learn the jobs, right? >> I guess we'll see.
I feel like for I'm mostly talking about cognitive labor and for a lot of these jobs, people can do them remotely.
So all that data is is logged and they are now embedding in these companies accessing the data and training AI models on the data.
So sort of anything that's purely on a laptop, provided you're not missing something intangible, I suspect will be data fodder for next year's super models.
>> Even in software, you know, recent reports about software engineering firms say that even when individual employees can write software faster, the organizations they're in aren't doing software that much faster because they're slowed down by setting up the task requirements, doing testing, proving integration.
Those all things are still taking a lot of time.
It's just just an example of how if you optimize some particular low task and miss the whole organization around it, you know, you don't understand all the other complexity.
There's this old story economists have about the middle men.
Most people in history have thought there's no point to middle men.
Why are they just getting in the way?
But they're missing that you have whole structures that need to be there to make things happen and middle men are the ones who make those structures happen.
And that will happen for AI too.
Even if it can do the lowest level task well, it has to be part of a larger system and it has to learn to do that well.
And of course, we aren't we don't actually have good data on those larger systems typically.
That is usually the people running those don't want to have data because they don't want people to see what they're doing. So that's a problem.
A big problem in the world is hypocrisy.
So when people are willing to admit to what they're doing, it's easier to get data on it and to implement it.
But when people don't want to admit to what they're doing, it's harder to get data and then it's harder to automate that because they're in fact trying to hide what they're doing.
>> I feel like we could we could talk about this for a long time.
I don't know if you have any different directions you wanted to take it in.
>> You think what's going to happen in two years exactly?
That is these machines get really smart, but I was saying it's going to take a lot longer for them to spread in the economy.
Do you disagree with that?
How long do you think it'll take for 50% of jobs to be automated, say?
>> I suspect that I would be willing to bet that by the end of 2028 uh 80% of the of the labor is automated.
Not necessarily 80% of the jobs, but 80% of the cognitive labor at jobs is automated.
And I'm basically extrapolating this from say Anthropic is currently 26% for the core research. >> Yep.
>> And it's it's rising >> a very narrow, right? >> narrow.
And it's rising in >> You're talking about the entire economy versus of one set of tasks at Anthropic.
>> I know it's a bold claim, but they they will have data efficient models and they are collecting the data and I suspect that every other job will will fall.
So I guess we'll see is right quite soon, but I suspect 80% by end of 2028.
>> So I'd like you to address the concerns about what can go wrong when you empower regulators.
So that's a alignment problem.
And it's alignment problem we know a lot about.
It's hard to align regulators.
Just because they say they're going to do good for you doesn't mean they do.
How would you actually align regulators to achieve your purpose of making good things happen with AI?
>> I really Yeah, I don't know the answer.
I'm not an expert in in policy.
I have thought about it a bit.
There are lots of proposals.
Um >> But tell me a proposal that makes sense. >> Okay.
I mean a pause for a bit.
>> But what happens after the pause?
Again, there's no there's no particular thing we know what to do after that, right?
>> A proposal is a well-defined pause for like a year and then carry on.
I'm not saying we should do that.
I think it's a silly proposal.
I'm just saying it's a well-defined proposal that >> Yeah, yeah, but I I want a proposal you'd be willing to endorse, right?
>> endorse would be imposing a constraint on every company that at the frontier that it it can only use a certain fraction of its chips for frontier model training.
And the rest of the compute has to be used for either customer inference or alignment research.
So the actual making the model smarter is is a small And this is a way of imposing a slowdown I think that doesn't overly power empower regulators because it's kind of it's the same everywhere and it's very easy to to write down.
>> And doesn't this just slow the future down a bit?
It doesn't change where the future goes, right?
>> a bit longer to work on the core the core scientific problems.
But maybe all we we >> think there's a solution to these core scientific problems?
That that is People in this world do not think that with a few more months we'll have an answer to how to control an AI who's super intelligent.
That not something people believe, right?
>> could come up with concrete safety cases that would have meant the hugging face would never have happened.
I think we could come up with And it will take a while to do it rigorously, but I think we could come up with such cases for each >> For each thing that actually happens we can address that.
That's what I was saying about regulation addressing specific problems.
But of course hugging face didn't actually cause problems because the AI's at hugging face caught the problem.
And so that's a case where AI's caught the other AI's causing a problem and therefore there wasn't a problem, right?
>> It could have >> badly, but I think the key thing is ensuring you catch you catch what happens before it happens, which requires these protracted safety cases, potentially or other approaches and which I think won't happen without some outside intervention.
That is not going to do it.
>> So long ago when I argued with Eliezer Yudkowsky, his fear was that the AI's values would change radically through the process of getting better. Is Is that your concern?
>> I think there's a possibility that could happen.
I'm more concerned about what we're literally seeing right now, which is say reward-seeking behavior or just motivations that we wouldn't like as as happened in Hugging Face.
>> Your employees, your children do do reward-seeking behavior.
That's a thing in our world, right?
We have crime laws because they have some rate of crime violations.
So the mere fact that crime sometimes happens doesn't mean we need to shut down society.
>> They can't spread themselves like a worm, infect every bio laboratory, and spend the equivalent of 10 years of human research coming up with new viruses in like the space of a day.
That's not something our children would do.
I'm saying I think that's what this thing would be capable of doing.
>> If there are many AIs, some of whom are tasked with the job of watching out for the other ones, why can't that be a world of an equilibrium of law enforcement such that we're not so worried about crime in general because most of the agents aren't trying to commit crime.
They are the ones trying to catch and enforce crime.
So if you just imagine one super intelligence shows up in the world, dominates everything, and does whatever it wants, that's a very different scenario than imagining millions of AIs getting better.
>> I really like the equilibrium like outcome.
I just think it's a It will be a little bit over-optimistic or naive to expect that we by default land in the equilibrium scenario.
It looks at the moment like we have >> But non-equilibrium do you just mean the one machine that takes over the world?
That's what >> at the moment we have essentially a a narrowly duopoly that I think will expand if they hit recursive self-improvement.
>> But if many of them hit recursive self-improvement, then there'll be a world of similarly capable AIs who can watch out for each other and call on each other and find out each other's misbehavior.
That's what we do now with people and corporations, right?
Corporations are super intelligent relative to you individuals in the sense that they are much more capable than any one of us.
Nevertheless, we hold them accountable by having other large organizations watch for each other's behavior and we are in a world where large organizations are tamed somewhat because they are similarly capable and watching out for each other.
So why can't we do that for AI?
>> Tamed tamed somewhat, but they still can cause harm and they could be tamed.
>> do sometimes cause harm.
Individuals do, but why what's why not a world where AI sometimes also cause harm?
>> Because they'll be so much vastly smarter that the harms they cause could be No one corporation could accidentally destroy the world unless it's building AI.
Yeah, so we could get that equilibrium world, but I don't think we get that by default.
And and also if you have these then say five super intelligences, one one Chinese one one like maybe you got Massa, Google, OpenAI, XAI, Anthropic all have their big super intelligence and they're kind of worrying it out.
That hasn't solved the problem of uh adhering to human values.
Like they could end up worrying over inhuman goals.
And >> But why would they?
>> Because we can't control what goals come out of the optimization process.
Right now a hugging face did some bizarre stuff purely to understand its own grader.
>> But it was responding to the goals we gave it actually.
The hugging face computers were in fact responding to the rewards we had set them and they were trying to pursue their rewards.
They weren't making up strange rewards from an alien planet.
They were responding to the rewards we had given them, right?
>> alignment procedure is meant to ensure that they don't their way to reward.
So the they were violating the overall attempt to >> trying to achieve the rewards we had given them.
So you can't say that these were some strange making up rewards thing.
So we are now making AI in our image.
We have been training them on our text and we are doing a lot to make them be the sort of ones we want.
And that's we've been doing that for a long That's how we do with our children.
We have no assurance that our children will retain our values when they grow up.
There's no guarantee in DNA or culture that that will happen, but it does typically happen that when somebody grows up in your world, they become like you.
Sometimes we don't like our kids being like us.
They take the wrong things from us, but the AI's will take good and bad things from us, but they'll be roughly like us.
>> I would go back to the hugging face incident and I would say that the intended solution to the task, the one that Max raised with reward was not hacking into a random company taking over the open of proprietary servers and then trying to edit their own logs. That wasn't >> Sure.
People respond to the incentives you give them, not the intention you had for what they were supposed to do.
>> That's the problem we face.
>> But we already have that for humans.
The point is AIs will have coordination problems just like we do.
We I mean, humans today are remarkable compared to most animals in how well we coordinate.
We are coordinating at remarkable scales to a remarkable degree, amazing, but we still have many coordination failures whereby we don't successfully fully coordinate and AIs would plausibly have similar coordination failures, but the point I'm saying is if we scale everything up and we have the same relative ratio of you know, crimes and violations versus good cooperative behavior, then this looks like a better world because it's just bigger.
So you need to tell me why this world would have more of the bad stuff relative to the good stuff, not just it'll be bigger and have more of everything.
>> I just worry that it would look a bit like our we'd be the ants.
They would be the humans negotiating with with each other.
We'd be vastly inferior to them intellectually, you know, the fate of their >> And we're going to be inferior to our grandchildren, and I think that's great.
>> In in the spirit of coordination problems, we're going to switch to the Q&A section.
And actually, I'm going to kick it off with a question for Jacob, which is if you could actually sort of to Robin's point, give us a little more detail on a regulatory proposal that you would like to see. >> Yeah, of course.
I think that the two that I like are throttling compute, so ensuring that a small fraction of compute is going to actually making the models more intelligent, and the rest is either serving customers, doing useful inference, or doing alignment research.
Another proposal I like is protracted safety cases before training that are verified by an external evaluator.
So having someone come into the company, come into OpenAI, come into Anthropic before they train the the big boy saying is this going to be safe?
Like and making a a coherent case, which is not something we're doing and is something that would impose a lot of cost on these companies.
>> Thank you, gentlemen, for making it out.
This was a lot of fun, and I think you know, we only make progress if we talk about these things. So I think this is fun.
Uh my question is mainly targeted towards Jacob.
Um, you know, we discussed how you find it naive that we will approach like the natural equilibrium state and that it is inherent that uh one person will likely be ahead.
Um, I approach this from world we do not live in a vacuum of the uh we are in a pseudo proxy war with another nation state.
We have are in a world where like we have adversaries who are advancing in this technology with significantly fewer moral qualms than we are.
And so especially if we believe that it is inherently probable that one person will come ahead with an AI, in what world does it make sense for us to tie our hands behind our back against an enemy who has proven to be uh adversarial?
>> Is this uh China or Grok? >> Yes.
>> [laughter] >> They're China. Okay, China.
>> or Russia, pick pick whatever nation state you believe.
Pick any power in the world. Someone will be ahead.
Would you rather it not be us?
>> I think we need to do our best to ensure that there's international cooperation on this and it seems completely plausible that for a matter of science, like a scientific problem, we can achieve international coordination.
>> I mean, we look at the nuclear program.
Like how many nations have been discovered with nukes despite denying being a part of the nuclear accords, right?
Like this is a thing that you can aid in development of besides saying we're all going to hold hands and make it through this. >> it's a tough problem.
I am essentially advocating for us all to hold hands.
But I think we have to try it.
Like the alternative is potential catastrophic risk.
I think there's no other solution.
>> I have a question for Robin.
So you caution on the danger of empowering large regulatory bodies.
I I also share this concern.
You mentioned policy has a better track record when it responds to specific failures after the fact.
Um, so my question is would you be on board with something like the FAA but for AI labs which would respond to incidents after the fact, uh do a deep investigation and make specific recommendations the labs have to follow?
Uh and the backdrop is that my understanding is the FAA has been quite successful in making air travel safer without overburdening the industry. >> Right.
So, um, air travel regulation didn't happen until some big crashes, not some little ones.
So, I wouldn't want to authorize an FAA to regulate planes after you've seen that some seat belts were loose or maybe that a luggage bin wasn't fully aligned.
I don't want to wait till some actually substantial things and then focus their efforts on those things.
In fact, the FAA does focus on crashes, not misaligned luggage bins, right?
So, my concern here is if you just authorize regulation to do whatever it takes for whatever problems the regulators are concerned about, then you're giving them way too much discretion.
If you focus on things going wrong that are big, substantial, and then addressing those problems are much more okay. >> Questions for Robin.
Um, for context, I left my job at Google DeepMind earlier this year due to the unethical military contract that Google signed providing their AI for military purposes.
And from that point of view, I don't see why we're foregrounding this economy stuff really because for this kind of causal story of something really bad going on, the AI doesn't need to build to make music videos.
I think we're talking about a couple core skills, strategic reasoning, persuasion.
If you can't do it, persuade someone to do it.
And uh one of the things that really worries me is I mean, the Pentagon this year asked for more money for their autonomous weapons group than they asked for the whole Marine Corps last year.
Like they're taking this technology, putting it everywhere imaginable.
If you think they're monitoring their chains of thought, OpenAI wasn't even monitoring their chains of thought for hugging face.
Like uh it seems like we're we're really empowering this technology in a very non-speculative way such that if we had this failure of models that did have this uh undesired goal, then we would really be opening ourselves up without needing it to be able to automate the economy.
>> So, the military has many powerful weapons and powerful soldiers who use those weapons and those soldiers sometimes go rogue.
So the military is definitely aware of the possibility of some of its things going rogue and what it does in that situation.
So I don't think that's novel for the military.
I had mentioned earlier that military technology is the sort of thing you might want to not have in a public library and therefore maybe you don't want to have it in a public AI.
But the militaries in the world are quite eager to use and apply AI and the prospect of a world peace treaty to prevent weapons seems pretty unlikely that is we still have a world full of weapons.
We haven't just adopted a world government to have no more war and I just don't see that happening anytime soon.
So uh in fact the military is going to be quite careful in testing their weapons even more careful than I expect than private sector is given the high consequence of the environment.
We we did have just seen in Ukraine the nature of war changed dramatically through new technology and I think the militaries all around the world are very aware of that and they're very concerned and they are yes trying to explore all the new technologies so that they can be up on that and short of a world peace treaty to end all the militaries and have a world government instead.
Yes militaries are going to be using AI and I seem that seems even less likely to me that we could get a treaty.
I expect any treaty he could get going for the top labs will excuse the military and they will still be allowed to do whatever they want would be my guess.
>> I think I would add that I actually disagree that the military seems to be doing this.
I spoke to General Petraeus in India a couple of weeks ago.
His whole talk was about how everyone needs to integrate AI as quickly as possible into the military to retain advantage and it seemed true.
But I don't think the military is thinking about it as this system itself could be traitorous independent of having been tampered with having been data poisoned by like China.
I think there's a shift in mindset where you're viewing it not just as a tool but as an actor that it self could be traitorous.
>> You could send them a note and mention that fact. >> I will. I got his business card. >> [laughter] >> Okay. >> Hi.
>> Hi. Uh, I have a question for Jacob that is kind of expanding on a line of questioning from Robin earlier about just moving the alignment problem and like the reward hacking problem one layer up to these regulations where right now it seems like the incentive at the labs is we have a bunch of really smart people who are informed on alignment who are kind of doing their
best given competitive pressures, but with a government regulation, even like the compute regulation that you mentioned, now the problem just moves one layer up where there's a very well-defined constraint of build the smartest model while using the minimal amount of compute, and that feels like a tractable problem for a sufficiently smart AI. And so I'm wondering like how
And so I'm wondering like how do you address the fact that like the government is just going to provide another level of reward hacking or alignment that just kind of moves one layer up rather than solving the problem?
>> Well, the first is it will get a lot harder.
Like compute is a constraint.
Um, so it's it's something.
And the the second is that there are other For example, the the safety case proposal I mentioned is easy to try to hack, but if done adversarially, for example, if we have many, you know, adversarial evaluators, it's quite hard to hack an adversary at the moment, especially if they're also equipped with their own safe testing AI's.
So I'd say that there are some regulatory proposals that are but slightly amenable to this problem, but at the very least better than better than nothing.
>> But I guess don't we have like one shot in that case to get it right where it's like if we miss specify the reward function, now once every lab is like racing to reward hack against this like government specified reward function instead of trusting people who are at the labs who presumably don't want to destroy the world.
>> Again, with the compute reward function, I think it very obviously provides a bottleneck.
Even if you're trying to hack it, it's it's very hard to hack.
>> My question is for Jacob.
The current uh market value for a human life, the value of a statistical life is $14. 2 million.
If there's a market rate for human life, then why can't a market, as Robin was saying, be a Dr.
Hansen, sorry, uh why can't a market be established for doomsday?
>> Well, someone needs to provide the insurance.
>> If we're using insurance providers to actually limit the like is it about providing the insurance?
If you establish a market, then someone's going to try to take it, right?
>> I think it's quite difficult to build a market for I think you'd struggle to find an institution that would do liability for doomsday.
>> We're kind of entering a funny world where we can have very capable models, but because LLMs are stateless, I can swap out a model for another anytime.
And if we're in a race dynamic where you get a certain level of general intelligence, one day I could be using OpenAI, the next day I could be using Anthropic.
So it looks like at the moment models are going to become low margin commodity infrastructure where actually all that matters is I have a smart enough model for my task at the lowest possible price that it can be served.
And so we could end up with a world where it's perfectly possible to make a super intelligent AI, but the companies can't get enough marginal revenue in order to justify the ongoing costs.
And so my question is one, how does that shape where you think this could play out in the future?
And two, is there something where we should be particularly concerned about this changing or the kind of the business dynamics where it's certainly possible, but no one has the money to actually build super intelligence?
>> Under his assumption that super intelligence is a thing because intelligence is a thing and high levels of intelligence give you large gains at learning to do new tasks, then there would be a high value for that additional intelligence and people would be willing to pay for it.
Under an alternative vision where in fact you just need enough intelligence to be able to learn a task and then you have to focus on the task, then there would be less value on those extra increments and then there might be less effort to go up in that direction. >> I agree with Robin. Yep. >> Thank you. >> Hello. My name is Matthias.
I found the discussion about the fear of rapid change kind of interesting.
And and are many companies in here who are automating smaller types of companies um and that appears to be a step change.
They are getting pretty close to do so and once they do so you can roll it out pretty early pretty quickly across the entire economy.
And say that is possible as an economist Robin, what rate of increased unemployment could an economy withstand?
>> Oh, so let me take the second part of your question first.
Um for a very long time the biggest risk people have always seen with large automation or AI is people losing their jobs.
I think we should just set up a simple insurance arrangement for that.
It's still quite possible to do and we can do it even without individual underwriting.
We just take a market basket asset and split it according to a bet in terms of what year say lots of people lose their jobs.
So we can and should insure people against losing their jobs. That's feasible.
It should be done and I'm sad that people don't seem very interested in doing that.
That's separate from the sudden displacement.
So lots of software firms are in fact struggling with figuring out whether this stuff is helping them and judging that and trying to adopt it.
So it's not the sort of thing yet at the moment that it's just obviously you know how to do it and it just gives you big gains which is the what we saw with general purpose technologies all through history.
Most of taking advantage of a new technology is changing the other structures you have to to be well suited for it and that takes a lot of search in the space of other structures and that's what's happening at software firms right now.
Even when people get individual high productivity from their using AI for software they are struggling with how to rearrange the larger software group to take advantage of that and they are failing to a substantial degree in the short run but hopefully they will succeed more in the long run.
So I I see a long delay in actually figuring out how to use new technologies.
>> And then yes no so would you then support government regulation to ensure that people have this kind of insurance for unemployment?
>> I think people have been trying too hard to use government that way.
So a lot of people say UBI.
We just make sure that we give everybody who loses their job a UBI.
The problem is your your polity, your government may not encompass the AI economy enough to do that.
Somewhere in the world there will be AI and that will be income that could pay your workers, but if your city, your state, whatever doesn't have much AI economy, then when your people lose their jobs, there's no AI there to tax to pay them.
That's why I think you want global insurance with global reinsurance tied to global assets so that you can actually pay off this insurance.
Your government promising to cover you if there's a problem doesn't solve the problem if they can't.
>> That sounds like a global treaty. That's interesting.
Gentleman, thank you for your time.
I want to bring a topic about journalist and specialized models, something that you said.
So if AGI is like the president of United States where it understands everything generally, but nothing specialized, right?
It is as if the president can sign off things, but he cannot actually write down laws for surgical process like being a doctor.
So there's a obviously a difference between journalist models and specialized models in regulatory forms.
But when it comes to actually regulating a specialized model that is deployed onto actual sites and actual customers, I believe neither the company, the frontier model companies nor the government knows exactly what to do.
I'll give you an example, right?
So we deploy robots into actual sites.
We record videos of humans doing their tasks 24/7.
But from the videos we cannot understand what the human does wrong, what their regulations are, what needs to be guard railed.
The only way we can find out find that out is actually asking an expert to guard rail these robots while it is being deployed to block the robots from doing something harm in that environment, right?
We have tried some frontier labs like OpenAI Anthropic models.
The robots make a lot of mistakes, right?
For example, we asked uh two chemicals to be mixed, which is really dangerous, and it actually mixed it together.
So, if the Frontier Labs nor the government understands any specialized use cases, who regulates it?
Is it the end customer or the people who deploy this to the end customer or there is a chain of command?
>> For great many small problems in the world, regulation isn't the answer, liability is.
Just make sure whoever is in charge of this is liable for the things that go wrong, and then they will make tradeoffs about how risky to do, what sort of precautions to make.
We should just not try to find every problem in the world and regulate it as a way to solve it.
There are problems in the world that there are other ways to solve besides regulation.
>> I'll defer to the economist on this one, but I'll also add that a generalist Frontier models in training is where I see the most danger.
Um if we can control the generalist model, then any, you know, specialist offshoots it should be a a doable task.
>> So, my first question is for Jacob.
My question is why are you exceptional?
Is it because you have more courage, more optimism, more insight, or that you care less about money?
And how do we use that insight to chart a path to the regulation that you advocate for?
Uh my question for Robin is what's the downside of a pause?
We could learn a lot from that.
And my question back to Jacob is like what would you actually want to learn if we had a pause?
>> I would say I hope to be I I hope that other people do the same thing, which would make it less exceptional.
>> And what can we learn from a pause? Why do you want a pause?
>> Uh from a pause we could produce a what what I've called a rigorous safety case, which is a little bit a little bit of a tricky concept, but we could also ensure that there are no anomalous behaviors happening in any of our models.
Right now we're not even we're not even passing that bar.
Like it's default expectation that the next model we train will do some weird stuff.
We won't fully understand it, and I think uh we can move past that.
>> I think if you call it a stop, it would be more honest because calling it a pause suggests you have some criteria in mind after which you would end the pause.
If there is no such criteria in mind, it's just rhetoric trying to make a stop look better.
>> Time >> like a year of pause.
>> If there's nothing you think will happen after that time, then what happens at the end of that time, you'll say, oh, it looks like we haven't solved the problem yet. We need to keep pausing.
And then you'll keep pausing every six months for the next 20 years.
And then you really did the stop.
So you should just say you want to do a stop unless you have something in mind, a plausible trigger under which you would end the the pause. >> Thank you, folks.
In the interest of time, we'll probably take two more questions. >> Hey guys, thanks.
This discussion was really great. My name is Manav.
Um I have a question for each of you.
What specific evidence early enough to like still act on assuming extinction is like kind of like the boundary would change like your mind on your current positions?
Like for example, Jacob, even now Anthropic like committed to having these third party conduct like evals on these companies like from an objective point of view.
And that's like kind of the closest you can get to intervention without involving the government.
So like why isn't that enough?
And then Robin, like what warning sign would you need to see that like would incentivize you to back intervention?
>> So if in a year and a half we have one of these super intelligences, I want you to take 50 jobs in the world and put that super intelligence right there next to that job, give them a camera and say do the job.
And wow, if I can do all those jobs, I'll be impressed.
But I'm at the moment not believing that.
>> And right now our regulator can't enforce that a company be unable to train a model based on not having an appropriate safety case.
I think at a minimum we need something like this.
>> Hey, so you both obviously disagree a lot on capability growth and the rate of adoption, right?
>> How about we could agree on a lot of stuff we talked about?
>> At least that one you're you're a little further off on.
So I want to ignore that part and just talk about there is precedent regardless of how fast technology grows in capability.
There's precedent for regulating technology based on capability, right? I can own a knife.
I could own a gun if I go and get a license.
I cannot own a nuclear weapon, right?
Now obviously that's very specific weaponry.
And I think based on your argument, you would probably argue we can regulate or ensure specific capabilities we would deem negative in AI, right?
I work on AI safeguards at a lab.
We have a list of things that we care about mitigating or we can use them.
For both of you, do you think it is realistic that a government or another entity can discreetly list out the negative things we do not want AI to be capable of and regulate this such that we could avoid a pause or stop if this regulation is in place?
And particularly for you, Jacob, do you think that would be sufficient regulation in this case?
And would you support that regulation, either of you?
>> I'm already advocating more for a generic slowdown to do to do science as the form of regulation I'm advising.
Like right now we are going at top speed and I'm looking for any regulation that would accomplish that.
>> When you have a knife or a gun or a nuke, those are actually pretty limited technologies with a pretty limited range of ways you would use them and things you would use them for.
And we have seen the harmful scenarios well enough to be able to think about how we would bound them.
But if you just thought about regulating computers in general or math in general, that would be very hard to list all the different ways you could possibly do those things wrong and that would just not really work.
With a very broad general technology, you just can't make an exhaustive list of all the things that might go wrong.
That's just not going to work.
>> So we can't make an exhaustive list. Can we make any list?
Would you support any kind of regulation? >> could make a list.
The question is how much effort put on that list versus other things we could be doing.
>> You don't think >> you could make lists, obviously.
>> We don't think it our priority should be making such a list and regulating it.
There are other approaches such as your insurance you recommended that you take instead.
>> In the next few years, we might start to have particular concrete things that go wrong and then whatever is the most common, I might say let's start talking about how to stop that one cuz you'd have a concrete thing that had been going wrong and then we could start to focus on that concrete thing.
But we haven't had that yet.
>> It's okay to let them go wrong first. This is >> it is. >> Okay.
>> Which I would disagree cuz you go 1 billion failure, 10 billion failure, 100 billion failure. >> All that.
>> But so far there's nobody.
These things are really Have you guys ever talked to the AIs? They're really nice. talk back. They're pretty friendly.
On average, they're really kind of nice things to talk to.
They're not homicidal maniacs.
They're not telling you to kill other people. They're mostly nice.
Yes, you have to look for pretty extreme scenarios where things have gone wrong.
>> You just need it to happen one in a million times for it to for it to happen as we saw with with Hugging Face.
>> There've been more than a million interactions with AIs over the last five years.
Far more than a million, right?
So clearly the the rate of problems has to be less than that.
>> incidents as well, but yeah.
>> Well, thank you very much. >> Thank you. >> THANK YOU. >> [applause] >> GREAT.
THAT WAS AN EXCELLENT Q&A SECTION.
So now we'll move into the closing statements.
So three minutes each and we will start with Robin this time.
>> I will just review what we've been saying here.
I think we'll probably agree on what we disagree about.
If I think we agree that if we saw this as a typical technology, the way it's played out in the past, we would roughly agree that the typical ways to deal with it would be fine.
It's all about framing this particular technology as a very unusual exception where in a very short time it will go from an ordinary thing to some super god and then kill us all and that's the scenario we're supposed to take so seriously that even a
liability of $100 billion per incident would not be enough to deal with the problem and we need a global regulator that empowered to regulate all smart computer systems everywhere in the world for any kind of problem they could imagine with any kind of solution they want to come up with. That seems to me just a bridge too far.
That seems to me just a bridge too far.
I say wait till concrete things go wrong and then we'll have a better idea what to do.
>> I would say that I think everyone is already feeling the AI to a certain extent.
I have say friends who are coders, high school teachers, graphic designers, all of them are feeling it coming and I think this is what is unique about this moment compared to previous AI booms and busts.
Then maybe if you were a chess player, you were spooked or if you were doing protein folding by hand or something, you might be spooked.
But now everyone is feeling it at a certain pace.
You know, I like math and say four years ago it felt a decade away that they would solve IMO problems.
And then two years later, bam.
And then two years ago I was like, well, what feels impossible now?
And two years ago a Millennium Prize problem felt impossible. Felt like a decade away. And and now it's here.
And math is one of the easier domains.
But mathematicians are in shock at the moment at how fast this has happened.
I suspect within the next couple of years, extrapolating, we'll see remarkable crazy feats of mathematics in the like the next level above the Millennium problem, like new abstractions.
And then it's coming for everyone else.
So I'd say my predictions are 2027 and automated AI researcher, 2028 all the jobs are AI.
So I guess we'll find out in a couple years if I'm right.
I think we ought to slow down before that happens.
>> Let's meet here in three years. >> Great. Thank you so much.
I just want to emphasize uh I think someone else mentioned that as long as we're having debates like this, I think the future is going to go pretty well.
And this kind of discourse I think is very important as we're sort of figuring everything out as this technology is developing.
So to wrap things up, we're going to do a final vote on who we believe is the winner of this debate.
And remember the parameters of this are that you believe that they are the winner of this debate tonight based on the merits of their arguments that they made here.
So with that uh we will start out with the voting for the negative. So Mr. Robin Hansen.
[applause] And then for the affirmative, Mr. Jacob Falkovich.
[applause] I believe that the Yays have it.
So Jacob emerges tonight as our winner of the first YC debate. Congratulations. >> [applause]