Dario Amodei sides with Sam Altman, says AI is risk for humanity, could go out of control

Anthropic CEO Dario Amodei warns that rapid AI progress could pose risks to humanity and calls for global cooperation to keep safety measures in step.

In Short

• Amodei warns AI could become a risk to humanity if poorly managed
• He calls for global cooperation and stronger AI safety standards
• Anthropic CEO urges safeguards against AI misuse and loss of control

After OpenAI CEO Sam Altman warned about the risks of uncontrolled AI, Anthropic CEO Dario Amodei has also told the UN Security Council that AI could become a threat to humanity if its rapid development is not properly managed. He argues that the risks are global and cannot be managed by AI companies alone, calling for greater cooperation between companies and governments to keep AI safety in step with its rapid development.

Speaking at the UN Security Council on Wednesday, Amodei said AI is advancing at an extraordinary pace. While the development is bringing major benefits to people, it is also creating risks around biological weapons and the possibility of humans losing control over increasingly capable AI systems.

“AI is advancing very quickly. Four years ago, it could barely write a line of code or solve a high school maths problem. Today, it writes most of the code at Anthropic and solves world-famous unsolved maths problems,” said Amodei. He believes this progress will continue rapidly, with AI becoming capable of doing the work of a very large number of highly skilled people. “The trajectory only needs to continue for a tiny period longer, one or two years, maybe less, to reach what I've called a country of geniuses in a data centre,” Amodei said.

While this rapid progress could bring major benefits, like the recent discovery of a new molecular machine by Claude, Amodei warns that the same capabilities also come with serious risks, and needs to be managed. “ I see two main categories of risk. The first is misuse of AI models. For example, misuse by bioterrorists to create biological weapons,”he said. “The second risk is loss of control. Model capabilities accelerating beyond developers' ability to control them.”

Amodei says these risks are not limited to AI companies or individual countries. He argues that they affect everyone and are serious enough to require the attention of the UN Security Council, rather than being treated only as an issue for economic or technology ministries. “If managed poorly, I even believe that AI could be a risk to humanity as a whole.”

Amodei highlights that his company Anthropic is already taking steps to manage these risks. The company tests its AI models for potentially dangerous capabilities, including in areas such as cybersecurity and biology, and puts stronger safeguards on its most capable models. It also publishes its safety research and risk assessments. He says Anthropic is also willing to slow down AI development if needed to make sure its systems are safe.

But Amodei argues that AI safety cannot be managed by individual companies alone and will require cooperation across the industry and between governments. He calls for countries to first agree on specific risks, including banning the use of AI to create biological weapons. He also urged governments to have systems to evaluate and verify the capabilities of advanced AI models. Alongside this, Anthropic boss also calls for common global standards to test AI systems for misuse and loss-of-control risks, as well as a system for countries to report AI incidents that could affect global security.

The warning from Amodei about the risks of advanced AI and the need for global cooperation came on the same day that OpenAI CEO Sam Altman also addressed the Security Council. Altman also warned that AI could bring major advances in medicine, education and productivity, but said humans could “lose control of the future to AI” if its development moves so quickly that people can no longer understand what is happening or intervene when needed.

- Ends