An AI hallucination nearly started a war between the US and China

The US military nearly attacked a Chinese vessel after a chatbot falsely claimed it carried nuclear weapons materials

Serving tech enthusiasts for over 25 years.
TechSpot means tech analysis and advice you can trust.

The takeaway: Warnings from AI developers have sparked widespread debate over whether, and how, the technology might cause human extinction. While much of that discussion has centered on rogue AI incidents, events in the Middle East suggest that military action driven by hallucinating AI tools may be the more immediate danger.

Sources connected to the US military recently told CNN that the US nearly moved to intercept a Chinese vessel in the Middle East this spring based on false intel from an AI-assisted report. The report was based on an inaccurate chatbot response that identified the ship as carrying materials for a nuclear weapons program.

Two sources said armed military personnel were preparing to board the vessel, with military planes already in the air, before officials took a closer look at the report and discovered it was at least partially AI-generated.

One source described the intel as completely false, saying it nearly started a war between the two nuclear powers. What was actually on the ship remains unclear.

An analyst had asked an unnamed chatbot to assess intelligence on the ship's manifest, information that originated with Special Operations Command Pacific. It's still unclear whether the chatbot was commercial or government-owned.

An official familiar with the US military's AI deployments told CNN, that a government chatbot would likely have been forked or cloned from a commercial product and augmented with government intelligence. Either way, the analyst also used AI to write an intelligence report based on the erroneous response – one that put the military on high alert.

The incident showcases the US military's push to accelerate AI adoption amid its war with Iran. In January, Defense Secretary Pete Hegseth announced plans to embrace AI more aggressively and eliminate bureaucratic barriers, warning that falling behind on the technology could cede the initiative to China and other US adversaries. Analysts, meanwhile, reportedly face mounting pressure to disseminate reports faster using AI.

// Related Stories

• The US just admitted, for the first time, that it has weapons deployed in space
• A government used AI to write its AI regulations. It did not go well

The military has primarily used AI for intelligence analysis, logistics, and targeting. Anthropic's Claude, for example, was central to the US's opening strikes in the Iran war, including the one that killed Iran's supreme leader, Ayatollah Ali Khamenei. But AI targeting based on outdated information may have also contributed to a strike on a school in Minab that killed 168 people, mostly children – an attack widely blamed on the US that UN-backed experts have flagged as a possible war crime.

US officials told CNN that the Chinese vessel incident isn't the only time the military has run into AI hallucinations.

See more TechSpot in Google Add us as a preferred source and our reporting shows up first when you search.
Add TechSpot