A new AI feature can control your computer to follow your orders

Who needs a mouse and keyboard when you have Claude?

News By Eric Hal Schwartz published 24 October 2024

An unseen, non-human hand moving the cursor across your computer screen and typing without using the keyboard in fiction is usually a sign of malicious AI hijacking something (or a friendly ghost helping you solve mysteries like the TV show Ghost Writer). Thanks to Anthropic's new computer use feature for its AI assistant Claude, there's a much more benevolent explanation now.

Fueled by an upgraded version of the Claude 3.5 Sonnet model, this AI – dubbed 'computer use' – lets you interact with your computer much like you would. It takes the AI assistant concept a step beyond text and a voice, with virtual hands typing, clicking, and otherwise manipulating your computer.

Anthropic bills computer use as a way for Claude to handle tedious tasks. It can help you fill out a form, search and organize information on your hard drive, and move information around. While OpenAI, Microsoft, and other developers have demonstrated similar ideas, Anthropic is the first to have a public feature, though it's still in beta.

"With computer use, we're trying something fundamentally new," Anthropic explained in a blog post. Instead of making specific tools to help Claude complete individual tasks, we're teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people."

The computer use feature is due to Claude 3.5 Sonnet's improved performance, particularly with digital tools and coding software. Though somewhat overshadowed by the spectacle of the computer use feature, Anthropic also debuted a new model called Claude 3.5 Haiku, a more advanced version of the lower-cost Anthropic model, though once capable of matching Anthropic's previous highest performing model, Claude 3 Opus, while still being much cheaper.

Claude | Computer use for automating operations - YouTube

Watch On

Invisible AI assistance

You can't just give an order and walk away, either. Claude's control of your computer has some technical troubles as well as deliberate constraints. On the technical side, Anthropic admitted Claude struggles with scrolling and zooming around a screen. That's because the AI interprets what's on your screen as a collection of screenshots, and then it tries to piece them together like a movie reel. Anything that happens too quickly or that changes perspective on the screen can flummox it. Still, Claude can do quite a lot by manipulating your computer, as seen above.

Unrestrained automation has obvious perils even when working perfectly, as so many sci-fi movies and books have explored. Claude isn't Skynet, but Anthropic has placed restraints on the AI for more prosaic reasons. For instance, there are guardrails stopping Claude from interacting with social media or any government websites. Registering domain names or posting content is not allowed without human control.

Get daily insight, inspiration and deals in your inbox

Sign up for breaking news, reviews, opinion, top tech deals, and more.

Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsors