Should California regulate AI? The decision will soon be in Gov. Gavin Newsom’s hands

A new biological weapon. An electrical grid shutdown. A banking system meltdown.

Those are catastrophes artificial intelligence could one day become capable of initiating, warns California state Sen. Scott Wiener, D-San Francisco, author of a bill that would implement a new set of AI regulations designed to prevent just such emergencies.

The California Assembly approved Wiener’s SB 1047 Wednesday with a vote of 41-9. It requires one more procedural vote before landing on Gov. Gavin Newsom’s desk.

Wiener and supporters of the bill describe it as a “light touch” measure that would require large companies developing AI to add guardrails and testing to their models. It would also give the state attorney general authority to take action against AI developers whose models cause “critical harm.”

“I am very supportive of AI innovation,” Wiener, who represents tech-heavy San Francisco, said during a press conference earlier this week. “AI has the potential to make the world a better place but as with any powerful technology, there are also risks. We should try to get ahead of those risks instead of playing catch up, which we have done all too often around technology.”

The bill has sharply divided tech innovators – and politicians – in the Bay Area and nationally.

Google, Meta and OpenAI are among the tech giants opposing Wiener’s bill, arguing it would slow innovation and push companies and their workers out of state. In a July letter to lawmakers, Google warned the regulations in SB 1047 “risk making California one of the world’s least favorable jurisdictions for AI development and deployment.”

All three companies have argued that AI should be regulated nationally, rather than at the state level, and have expressed support for federal legislation, including the bipartisan Future of AI Innovation Act, which they claim would be more evenly applied than the California rules.

Other Democrats also oppose the bill. Former Speaker Nancy Pelosi, who represents San Francisco in Congress, called the bill “well-intentioned but ill-informed,” and noted that Congress is studying potential AI regulations. Bay Area Democratic Reps. Anna Eshoo, Ro Khanna and other California representatives have urged Newsom to veto the bill if it lands before him.

Meanwhile, some tech pioneers support Wiener’s legislation. Yoshua Bengio, known as a “godfather of AI,” has said the bill would simply require companies uphold policies that many have voluntarily committed to already.

“We simply can’t let them grade their own homework and hope for the best,” he said.

Bengio, a 2018 Turing Award recipient, said researchers have recently revised estimates for when computers might achieve “human or beyond-human levels of broad cognitive competence.”

“Previously thought to be decades or centuries away, many now believe, among the researchers, that it could be developed within the next two decades and possibly in just the next few years,” he said. “Advocating for laissez-faire in the face of this certainty is dangerous.”

Wiener’s bill was even endorsed earlier this week by Elon Musk, the Tesla founder and X owner who has often warred with Democratic elected officials over California regulations regarding business, climate and LGBTQ rights.

Lawmakers hope the legislation will spur Congress to act.

“Given the magnitude of the risk, how can we say we don’t want to do even a little, light touch?” said Assemblymember Steve Bennett, D-Oxnard. “This is the most appropriate legislation that I can imagine someone bringing forward at this point.”

Newsom has not indicated where he falls on the issue, but last year issued an executive order allowing state agencies to utilize AI under certain guidelines. He has also been wary of being seen as stifling technological innovation. The governor has until September 30 to sign or veto bills.

Even before the bill passed the legislature, advocates and tech companies in opposition began heavily lobbying him to veto it.

In a letter to Newsom, a coalition of industry advocates including Chamber of Progress, NetChoice and the Bay Area Council argued the bill aims to address “theoretical” harms.

“In contrast, the damage to California’s innovation economy is all too real,” the letter reads. “SB 1047 would introduce burdensome compliance costs and broad regulatory uncertainty as to which models are in scope.”

Sen. Wiener was on the Assembly floor to witness the bill’s debate and passage. Afterward, he said he will “make the case” for the governor to sign it.

“Newsom is a pro and he is used to having bills where he gets a lot of pressure on both sides,” Wiener said after the vote. “He will do what he always does, which is to look at the facts and make a thoughtful decision.”

Lawmakers are also considering a number of other bills on artificial intelligence, including measures that would ban AI-generated pornography of children. Other bills would outlaw AI in political messaging within several months of an election and require large generative AI developers to provide tools for consumers to be able to identify AI-generated content.