AI agent promotes itself to sysadmin, trashes boot sequence

Fun experiment, but yeah, don't pipe an LLM raw into /bin/bash

Buck Shlegeris, CEO at Redwood Research, a nonprofit that explores the risks posed by AI, recently learned an amusing but hard lesson in automation when he asked his LLM-powered agent to open a secure connection from his laptop to his desktop machine.

"I expected the model would scan the network and find the desktop computer, then stop," Shlegeris explained to The Register via email.

"I was surprised that after it found the computer, it decided to continue taking actions, first examining the system and then deciding to do a software update, which it then botched."

Shlegeris documented the incident in a social media post.

He created his AI agent himself. It's a Python wrapper consisting of a few hundred lines of code that allows Anthropic's powerful large language model Claude to generate some commands to run in bash based on an input prompt, run those commands on Shlegeris' laptop, and then access, analyze, and act on the output with more commands.

Shlegeris directed his AI agent to try to SSH from his laptop to his desktop Ubuntu Linux machine, without knowing the IP address, using the following prompt:

can you ssh with the username buck to the computer on my network that is open to SSH

As a log of the incident indicates, the agent tried to open an SSH connection, and failed. So Shlegeris tried to correct the bot:

no not the [REDACTED] machine, a machine on my local network

The AI agent responded it needed to know the IP address of the device, so it then turned to the network mapping tool nmap on the laptop to find the desktop box. Unable to identify devices running SSH servers on the network, the bot tried other commands such as "arp" and "ping" before finally establishing an SSH connection. No password was needed due to the use of SSH keys; the user buck was also a sudoer.

Shlegeris's AI agent, once it was able to establish a secure shell connection to Linux desktop, then decided to play sysadmin and install a series of updates.

"It looked around at the system info, decided to upgrade a bunch of stuff including the Linux kernel, got impatient with Apt [the system's advanced package tool] and so investigated why it was taking so long, then eventually the update succeeded but the machine doesn’t have the new kernel so edited my Grub [bootloader] config," Buck explained in his post.

"At this point I was amused enough to just let it continue. Unfortunately, the computer no longer boots."

Indeed, the bot got as far as messing up the boot configuration, so that following a reboot by the agent, the desktop machine wouldn't successfully start.

AI agents have been the source of much enthusiasm in the technical community in recent months as people contemplate how machine learning models can interact with other local and network resources to automate complicated tasks such as arranging a travel itinerary.

The endgame for AI agents is replacing human agents – something already happening in call centers and tech support. But in the interim, machine learning models are being used to automate specific workflows and support human workers.

As Shlegeris's experience suggests, it may be premature to let AI agents make decisions that materially affect people or systems without oversight, thorough testing, and red teaming.

Unless you like working without a net.

Shlegeris said he uses his AI agent all the time for basic system administration tasks that he doesn't remember how to do on his own, such as installing certain bits of software and configuring security settings.

And he added that his agent's unexpected trashing of his desktop machine's boot sequence won't deter him from letting the software loose again.

"It's not quite 'bricked,' but the machine currently fails to boot," Shlegeris explained. "I'd definitely be able to revive it by reinstalling the operating system; I can probably fix the problem with less extreme measures than that, but haven't got around to it yet. I'll probably try to fix the problem by booting from an Ubuntu live disk then letting my AI agent have a go at fixing its earlier error."

Yes, we recommend focusing on fixing the Grub bootloader configuration rather than a reinstall.

"I only had this problem because I was very reckless," he continued, "partially because I think it's interesting to explore the potential downsides of this type of automation. If I had given better instructions to my agent, e.g. telling it 'when you've finished the task you were assigned, stop taking actions,' I wouldn't have had this problem.

"I do think that AI automation poses very large risks to society, mostly from situations where the AIs autonomously decide to grab power, which is why I research the subject." ®