0:00
The rocks can talk, but they can also read, they can see, and now they can use a computer.
Search this video
0:07
Browsing the web, clicking buttons, typing text all by itself.
0:11
The age of AI agents is here.
0:15
One of the first out of the gates is Claude Computer Use, Anthropic's brand new AI agent.
0:21
Let's dive into how it works, what it can do, and how it may change AI forever.
0:34
In October, Anthropic made waves when it released a set of upgraded models, Claude 3. 5 Haiku and a new 3. 5 Sonnet.
0:44
They also released something special, Computer Use.
0:47
But they're not the only ones in this space.
0:49
We already know Sam Altman is working to recreate Samantha from the movie Her.
0:55
And OpenAI is said to be releasing its own agent, Operator, in the new year.
1:00
Google is working on something similar, too.
1:03
The landscape for AI agents is growing fast and so far, Anthropic is the first of the big AI labs to get into the game.
1:11
Right now, Claude Computer Use is still in public beta as developers put it to the test.
1:17
But already, it's looking like a complete game changer. So, how does it work?
1:23
Claude had the ability to understand images for a while, so the next step was to train it on how and when to perform specific actions, like clicking buttons or writing text, based on what's displayed on the screen.
1:37
Claude has, has had for a long time, since Claude 3 back in March, the ability to analyze images and respond to them with text.
1:46
The only new thing we added is, those images can be screenshots of a computer.
1:51
And in response, we train the model to give a location on the screen where you can click and or buttons on the keyboard, you can press in order to take action.
2:01
And it turns out that with actually not all that much additional training, the models can get quite good at that task.
2:08
It's a good example of generalization.
2:10
For this, Anthropic needed to train Claude to recognize exact locations on the screen, down to the pixel.
2:18
Anthropic was then able to train Claude to understand what's happening on screen, and to reason about how it should use its software tools to do tasks.
2:27
For example, it might help you automate boring and repetitive tasks.
2:32
Claude's going to start taking screenshots of my screen and quickly realizes that the Ant Equipment Company isn't actually in the spreadsheet.
2:38
Luckily, we get a search match and Claude then starts scrolling through the page looking for all the information it needs to fill out this form.
2:46
To get started with Computer Use, developers have to run it in a virtual machine or container like Docker.
2:51
You'll also need an Anthropic API key.
2:53
Once that's all set, you can then open a dedicated browser window which shows the user prompt on the left and Claude's activity on the right.
3:02
Claude starts by analyzing the prompt and deciding which tool to use.
3:06
As it works, it takes a screenshot at each step to check its progress, making sure the task is on track.
3:13
If adjustments are needed, Claude loops back to try different actions or tools until it completes the task.
3:20
This repeatable loop of deciding, evaluating, and acting is called the "agent loop" and it's how Claude handles complicated step-by-step tasks all on its own.
3:30
So, what else can Computer Use make possible?
3:34
In their own demos, Anthropic shows us a few different tasks.
3:38
Like this one of Claude helping to plan a sunrise hike at the Golden Gate Bridge.
3:43
It searches the web, figures out some important details, and then creates an event in Google Calendar.
3:51
In another example, Wharton professor Ethan Mollick puts Claude Computer Use to the test by feeding it a video of a construction site and prompting Claude to monitor the site and look for issues with safety.
4:05
You'll see Claude take screenshot after screenshot analyzing different parts of the site, making note of all the gear and materials, and trying to spot any potential issues.
4:16
It even finishes up by putting everything together in a nice neat spreadsheet.
4:21
Automated OSHA compliance? Check.
4:24
By now, it should be clear that Computer Use is a step forward for AI.
4:28
Up until now, developers have had to make tools to fit the model, coming up with custom environments where AIs use specially designed tools to do different various tasks.
4:39
Now, we can make the model fit the tools.
4:43
That's a powerful change.
4:46
Computer Use opens up so many applications.
4:49
Businesses can automate repetitive tasks and increase efficiency, while the average user can save time on routine things like booking flights or ordering food.
4:59
It's easy to see a future where AI agents handle most of the drudge work for us.
5:05
And for developers, Computer Use massively lowers the barriers to entry.
5:09
LLMs have already made tasks like coding way more accessible to the average person, and Computer Use takes that a whole step further.
5:20
Computer Use is still a work in progress, so it has some bugs and limitations.
5:24
It's much slower than typical models and has a tendency to crash from time to time.
5:30
So, reliability is still an early concern.
5:34
Occasionally, Claude will misstep in its tool selection, get confused, or even sometimes veer off task.
5:41
During one session that Anthropic shared on YouTube, Claude unexplainably started searching for pictures of Yellowstone National Park out of nowhere in the middle of its task.
5:53
To be fair, humans get distracted and sometimes do that too.
5:56
Claude does have guardrails.
5:58
Since it could easily be used for abuse, it steers clear of things like account creation or content generation for social media.
6:06
It's also vulnerable to prompt injection, a security risk where the model can be tricked to follow different information or prompts embedded in the online sources it visits, rather than sticking to the original prompt.
6:21
Imagine a website prompt injecting Claude to upload the contents of your password manager. That'd be bad.
6:28
Anthropic thought about this and tries to keep users safe by keeping actions contained to a secure virtual machine, limiting access to sensitive data, and strictly controlling approved sites.
6:40
However, many of these limitations could be lifted soon, because this beta is just the beginning.
6:46
Anthropic already said that Computer Use will rapidly improve to become faster, more reliable, and more useful for the tasks users want to complete.
6:57
Plenty of startups are getting into the mix too.
7:00
Just recently, a YC company, Kura, released their own browser agents that seem to outperform Claude Computer Use on the WebVoyager benchmark, achieving a new state of the art.
7:11
In the near future, LLMs, with the full ability to use and control computers, will reshape everything.
7:18
How developers write software, how CEOs run their companies, and even how we all live our daily lives.
7:27
Each new groundbreaking application will transform how we work, connect, and live.
7:33
This kind of AI won't just be an assistant, it'll take on entire tasks that once needed whole teams or companies.
7:43
So, what will you build with Computer Use?