from the doesn't-anyone-check-anything? dept

South Africa Used AI To Write Its Now Withdrawn AI Policy. The Citations Were Fake.

Given how often we’ve seen AI-generated fake citations show up in legal filings and even legal decisions, you’d think the lesson would have sunk in by now: if you’re going to use AI to help draft something, you have to actually check what it produces. Apparently that lesson has not reached every government ministry.

Researcher Damien Charlotin was hunting for hallucinated citations using software he’d built for exactly that purpose, when he flagged something worth pausing on: South Africa’s proposed national AI policy contained at least four citations that don’t appear to exist.

The policy that contained hallucinated citations was, in part, a policy about the dangers of AI-generated misinformation.

And, days later, South Africa withdrew the proposal entirely.

South Africa has withdrawn its first draft national AI policy after revelations that it ​contained fictitious sources in its reference list ‌which appeared to have been AI-generated.

“The most plausible explanation is that AI-generated citations were included without proper verification. ​This should not have happened,” Minister of ​Communications and Digital Technologies Solly Malatsi said.

“This ⁠failure is not a mere technical issue but ​has compromised the integrity and credibility of the ​draft policy,” he wrote in a post on X on Sunday.

Compromised the integrity and credibility of the policy? Bit of an understatement, I’d say.

And, look, it’s perhaps no surprise that those looking to put in place an AI policy would be using the tech themselves, but it’s difficult to think that they can regulate it well when they don’t even appear to understand how to use it well (and when not to use it at all).

Naturally, the minister’s takeaway is that the tech needs more regulation:

“This unacceptable lapse proves why vigilant human ​oversight over the use of ​artificial intelligence is critical. It’s a lesson we take with humility,” ​he wrote.

That really feels a lot like blaming the tech for humans making dumb decisions with the tech. He’s not wrong that we need human oversight of the tool. The power of AI tools is only recognized when they are there to assist humans, not replace them, but it’s not clear how a policy position fixes that.

To me, this is more evidence that we need to do a much better job educating people about what these tools can and can’t do. And that’s harder than it sounds, because the companies selling these products have spent years aggressively overselling what AI can do while burying the caveats about how it should actually be used. The gap between what vendors promise and what the tools actually deliver is a big part of why people keep reaching for them in exactly the wrong contexts.

Malatsi’s instinct — regulate harder — is understandable, but it addresses the wrong problem. The behavior you’re trying to regulate here isn’t malicious; it’s lazy and uninformed. Regulation is reasonably good at deterring bad intent. It has a much worse track record against ignorance. People are going to keep trying to force these tools to do things they’re not good at, regardless of what the rules say, because convenience and overconfidence are powerful forces. The better outcome comes when people learn, through repeated direct experience, that the tool fails in these situations — and when the companies selling these tools are honest about where they fail.

There are still genuinely useful ways to deploy AI, even if stories like this make people think that the tech is never good at anything. But using it to generate citations for official government policy documents, without verifying a single one, is not among them.

Of course, rather than actually dealing with any of this, expect a new crop of startups offering tools that claim to review your AI-generated content for hallucinated citations — and are just as unreliable.