Nancy Pelosi’s replacement is ready to regulate AI

As a California state senator, Scott Wiener authored legislation aimed at curbing the risks of advanced AI. Now, after winning a tight congressional race for former Speaker of the House Nancy Pelosi’s open San Francisco seat, he could play a key role in setting federal AI policy. 

I spoke with Wiener on Friday, only a few hours after he had been declared the victor. But he was already thinking about AI risk. The previous day, the AI nonprofit METR had released a report that had been widely interpreted as evidence that Anthropic had achieved recursive self-improvement, a phenomenon in which AI meaningfully contributes to its own advancement and AI progress potentially spins out of control.

“It’s very concerning,” Wiener said, citing the report as evidence that we are quickly running out of time to regulate AI. “It was my hope that Congress would take decisive action before [the end of] this year,” he said. “That does not appear to be happening.”

Wiener has been increasingly focused on AI in recent years, and he’s made many of the AI safety movement’s concerns his own: During our 20-minute conversation, he used the phrase “recursive self-improvement” half a dozen times, for example.“We need to slow this down,” he says. “Because it’s moving very quickly, and we could sleepwalk into a disaster.”

Wiener has already had real influence on AI policy. SB 1047, a bill that he authored in 2024, was set to become the first real piece of frontier AI legislation in the country before it was vetoed by California governor Gavin Newsom; he then sponsored SB 53, a lighter-touch bill that Newsom signed. Both bills have inspired proposed federal legislation. 

Wiener’s congressional campaign received over $3 million from Public First Action, a nonprofit in favor of AI regulation that receives substantial funding from Anthropic. That contribution led opponents to criticize him for being too close to the AI industry, but Wiener says that couldn’t be further from the truth. 

“Public First supported me because I was the first lawmaker in the country to introduce AI regulation and the first to pass it and get it implemented,” he wrote in a message after our interview. “The attacks on me for [Public First’s] support were shockingly cynical—literally attacking a candidate who actually regulated AI over Big Tech’s objection and falsely claiming the candidate is somehow too close to Big Tech.”

Wiener says that if Congress forms a select committee on artificial intelligence—something that Democrats have discussed since September—he’d be eager to serve. And he’s enthusiastic about many of the policy ideas floating around Congress. “I think it’s really important to have a strong, empowered, nimble federal regulator,” he says, noting that Representative Ro Khanna, a fellow Democrat from California, has proposed legislation to establish such a regulator. 

He also highlights the importance of whistleblower protections, testing and auditing requirements, and a kill switch requirement, and he wants to see Congress leverage its oversight powers to hold the AI labs to account. “This is about the future of humanity, and we want to make sure that the public has a say and that the public knows what's going on,” he says.

And as a longtime state legislator, he is opposed to broad federal preemption of state AI laws, which some of the proposed federal legislation would require. “States are, as Brandeis says, laboratories,” he notes. “You want the states to be able to continue to be nimble in ways that Congress sometimes is not.”

Above all, Wiener says, he wants Congress to pass policies with real teeth. “It’s critically important that when Congress acts, that we do so in a way that is truly meaningful and that meets the moment,” he says. That means taking seriously the possibility of AI catastrophe, something he discusses in the sort of language typically used by AI safety experts. “We’re in a moment that is very dangerous in terms of the risk of loss of control over AI,” he told me. “We need to deal with recursive self-improvement. Once you’re in that realm, that really is potentially a recipe for total loss of control.”

He’ll have some allies in the fight against catastrophic AI risk. Senator Bernie Sanders, for instance, has met with many of the AI safety movement’s most influential figures and has spoken repeatedly about AI and existential risk. But there’s still a large gap between Wiener’s Bay Area, AGI-pilled perspective and the national mood on AI, which is anchored by opposition to data centers. The fact that voters approved 40 out of 41 anti-data-center ballot measures on Tuesday might seem to be a good sign for Wiener and others who support AI regulation, but Wiener says he sees this movement as far more tied to local concerns than to bigger questions about AI risk.

“Peoople who are really opposed to where AI is going—this is an area where they can actually do something about it,” he says. “But the data center revolt—to me, I think it’s more so that people are upset that there could be something like this in their community, with all the impacts near people’s homes, driving up electric bills and using water. I think that’s the bigger issue for most people, from what I can tell.”

Wiener describes himself as only “cautiously optimistic” about Congress’s ability to rein in AI, given its poor track record on regulating the tech industry. He also has concerns about the ability to get legislation past President Donald Trump, who called concerns about catastrophic AI risk a “hoax” in September and has demonstrated a stark unwillingness to collaborate with China, something that Wiener sees as essential to preventing the worst consequences of AI.

But he says Trump’s unpredictability provides  some measure of hope, though it’s a fraught kind of hope. “So far what we’ve seen are some problematic things, but not complete disasters,” Wiener says. “I hope that it doesn’t take a disaster to spur the president into acting. And I hope if that’s what it does take—which, like I said, I hope it doesn’t—but if it does, I hope it’s as small as possible.”