The bill, SB 1047, was introduced seven months ago by Sen. Scott Weiner and passed the state Assembly on Wednesday.
The bill now heads to Gov. Gavin Newsom's desk, who has until Sept. 30 to either sign it into law or veto it. The governor has not publicly stated his position.
The bill aims to require companies that spend more than $100 million to train AI models to develop safeguards to ensure the technology isn't used to harm society, such as building dangerous weapons or carrying out cyberattacks.
The measures include requiring companies doing business in California to report any safety incidents to the government, protecting whistleblowers, allowing third parties to test the safety of models, and, if necessary, forcing companies to shut down entirely.
Team Selection
The bill draws a line in Silicon Valley, with some of the tech industry's biggest names taking sides.
OpenAI chief strategy officer Jason Kwon warned in a letter to Sen. Wiener on Wednesday that the bill could stifle progress and drive companies out of California.
The creators of ChatGPT joined lobbying against the bill with leading company Meta, which said the bill could stifle the open source movement by imposing significant legal liability on developers.
A former OpenAI employee said the company's opposition to the bill was disappointing but consistent with the company's recent policies.
“We joined OpenAI because we wanted to ensure the safety of the incredibly powerful AI systems they were developing,” former OpenAI researchers William Sanders and Daniel Kokotajiro wrote in the letter, “but we are leaving OpenAI because we have lost confidence that the company will develop AI systems safely, honestly, and responsibly.”
Elon Musk also supported the bill.
Musk, who founded AI company xAI last year and is a longtime rival of OpenAI's Altman, said in a social media post on Monday that while it was a “tough call and it will anger some people,” he believes “California should probably pass SB 1047, the AI Safety Bill.”
“For over 20 years, I have advocated for regulating AI, just as we would regulate any product or technology that poses potential risks to the public,” Musk wrote on X.
Amazon-backed Anthropique appears to have changed its position midway through the debate after the bill was amended.
In a letter to California Gov. Gavin Newsom last week, Anthropik CEO Dario Amodei said the bill's “benefits likely outweigh the costs,” but added that he was “not so sure and there are several aspects of the bill that still seem concerning or ambiguous to me.”
Amodei said the bill at this point “seems to be somewhere between what we proposed and the original bill.”
