SACRAMENTO, Calif. (AP) — An effort to establish safeguards for the nation's first and largest artificial intelligence system passed in California in a key vote Wednesday, potentially paving the way for U.S. regulation of the rapidly evolving technology.
The proposal aims to mitigate potential risks posed by AI by requiring companies to test their models and publish safety protocols to prevent them from being manipulated to, say, take down a state's power grid or help create chemical weapons — a scenario that experts say could play out in the future given the industry's rapid advances.
The bill is one of hundreds lawmakers will vote on in the final week of the session. Gov. Gavin Newsom has until the end of September to decide whether to sign the bills into law, veto them or allow them to take effect without his signature.
The bill narrowly passed the Assembly on Wednesday but needs a final vote in the Senate before it reaches the governor's desk.
Supporters said the bill would establish some of the first long-awaited safety standards for large-scale AI models in the U.S. The bill targets systems that require more than $100 million in data for training — a bar that no current AI model meets.
“It's time for big tech companies to follow some rules – not many, but some,” Republican Rep. Devon Mathis said Wednesday in support of the bill. “The last thing we need is an outage on our power grid or water system.”
The proposal, authored by Democratic Sen. Scott Wiener, faced fierce opposition from venture capital firms and technology companies, including Open AI, Google and Meta, the parent company of Facebook and Instagram, which argue that safety regulations should be established by the federal government and that California's law targets developers, not those who use and misuse AI systems.
Several California congresswomen also oppose the bill, with former House Speaker Nancy Pelosi calling it “well-intentioned but ill-informed.”
The Congress for Progress, a left-leaning trade group funded by Silicon Valley, said the bill was “based on science fiction fantasies of what AI will be like.”
“This bill has more in common with Blade Runner or The Terminator than it does with the real world,” senior technology policy director Todd O'Boyle said in a statement after Wednesday's vote. “A key economic sector in California should not be thwarted because of a theoretical scenario.”
The bill is backed by Anthropic, an AI startup backed by Amazon and Google. Wiener amended the bill earlier this month to include some of the company's proposals. The current version of the bill removes perjury penalties, limits state attorneys general's power to sue violators and narrows the accountability of the new AI regulator. Elon Musk, owner of the social media platform X, also voiced his support for the proposal this week.
In a letter to Governor Newsom, Anthropik said the bill is crucial to preventing devastating misuse of powerful AI systems, and that its “benefits likely outweigh the costs.”
Wiener said his bill took a “light touch” approach.
“Innovation and safety can go hand in hand, and California is leading the way,” Governor Weiner said in a statement after the vote.
He also slammed critics who earlier this week dismissed the potential catastrophic risks from powerful AI models as unrealistic, saying: “If they really think the risks are bogus, then they should have no problem with this bill.”
Wiener's proposal is one of dozens of AI bills introduced by the California Legislature this year aimed at building public trust, fighting algorithmic discrimination and banning deep fakes related to elections and pornography. As AI increasingly influences Americans' daily lives, state lawmakers have sought to strike a balance between curbing the technology and its potential risks without stifling a burgeoning local industry.
California, home to 35 of the world's top 50 AI companies, has been an early adopter of AI technology and could soon be able to deploy generative AI tools to address things like highway congestion and road safety.
Governor Newsom declined to comment on the bill this summer but warned against over-regulating AI.
