According to Reuters, the California State Assembly has passed the Safe and Secure Innovation for Cutting-Edge Artificial Intelligence Models Act (SB 1047), one of the first major regulations on artificial intelligence in the United States.
The bill, which has become a flashpoint in Silicon Valley and beyond, would require AI companies operating in California to take several precautions before training advanced underlying models, including being able to shut down the models quickly and completely, ensuring that models are protected from “unsafe post-training modifications,” and maintaining testing procedures to assess whether the model or its derivatives pose a particularly high risk of “causing or enabling significant harm.”
Senator Scott Wiener, the bill's lead author, said SB 1047 is a very reasonable bill that would require large AI labs to do what they've already committed to: test large-scale models for catastrophic safety risks. “We've been working hard all year to refine and improve this bill, along with open source advocates, Anthropic and others. SB 1047 aligns well with what we know about foreseeable AI risks and is worthy of being enacted.”
Critics of SB 1047 (including OpenAI, Anthropic, politicians Zoe Lofgren and Nancy Pelosi, and the California Chamber of Commerce) argue that the bill places too much emphasis on catastrophic harm and could disproportionately harm smaller open source AI developers. In response, the bill was amended to replace potential criminal penalties with civil penalties, narrow the enforcement powers given to the California Attorney General, and adjust requirements for participation in the “Frontier Models Commission” established by the bill.
After the state Senate votes on the revised bill, which is expected to pass, the AI Safety Act will be sent to Governor Gavin Newsom, who will decide its fate by the end of September, according to The New York Times.
Anthropik declined to comment other than to point to a letter its CEO, Dario Amodei, sent to Newsom last week. OpenAI did not immediately respond to a request for comment.
