Anthropic AI safety bill would let regulators pause risky models

The draft, tentatively called the Artificial Intelligence Evaluation, Governance, and Incident Safety Act (AEGIS for short), sets a very high bar before the government can step in. A model could be paused only when it poses a foreseeable and imminent risk of catastrophic harm, measured by those two numbers: more than 50 people killed or badly hurt, or over $1 billion in losses.

The danger also has to come from one of three places:

• a cyberattack carried out with the model’s help;
• the model slipping out of its developer’s control;
• the creation or release of a biological weapon.

Those thresholds closely track California’s SB 53, the state’s frontier AI law. An Anthropic spokesperson said the bill puts into legal form the company’s Advanced AI Framework, which it has pitched to federal and state lawmakers since June, and called its proposals “sensible policies” for Congress.

Beyond the pause power, the draft would require large AI labs to set up formal safety frameworks, publish transparency reports and system cards for new models, protect whistleblowers and report serious safety incidents to the government.

Only one agency could ask for a pause: the Commerce Department. Even then it couldn’t act alone. Commerce would have to take its evidence to a federal judge in Washington, DC, who would decide whether to order a model’s release or use halted. The draft explicitly bars every other federal department, agency or official from suspending a model’s development, deployment or internal use.

Checking the labs would fall to outside evaluators. They would register with Commerce and periodically review each covered company and its models. Labs could pay their own evaluators, as long as the fee didn’t depend on the result. If an evaluator spotted an imminent risk of the listed kind, it would have 72 hours after finishing its review to tell Commerce.

Not every AI company would be covered. The draft targets developers that train very large models, measured by the computing power used, and that also either earn more than $500 million a year from AI or spend over $1 billion a year on AI research.

Critics say the bill is weaker than what Congress is already weighing. The main comparison is the FRONTIER Act, a bipartisan House bill from Reps. Jay Obernolte (R-Calif.) and Lori Trahan (D-Mass.) introduced in July, which would also override some state laws.

Their objections fall into a few groups:

• No licensing for auditors. The draft sets no standards for evaluators. Charlie Bullock of the Institute for Law & AI warned that, without licensing, auditors could compete to offer the lightest checks at the lowest price, even though he called the overall plan “pretty reasonable” and more than other labs have offered.
• A harder path to blocking a model. Nathan Calvin, general counsel at the advocacy group Encode, argued the proposal could leave courts with less power to stop a dangerous model than they have under current law.
• States lose ground. The bill would cancel some state rules on risk disclosures, outside testing and incident reporting, while leaving child-safety and civil-rights laws alone. In June, Anthropic itself wrote that states should keep full authority to legislate absent a strong federal law. State Senator Scott Wiener, who wrote SB 53, said the plan “falls far short” on public safety.
• An outdated size test. Others argue the computing threshold that defines a frontier model is set so low that it now catches most enterprise-grade models, though the revenue and spending tests narrow who is actually covered.

Anthropic didn’t defend every line. Anthony Cimino, who leads the company’s federal affairs team, conceded that the language on overriding state laws was “not precise enough” and said it will be redrafted. He said states should help set evaluator standards and help decide when a lab must shut a model down. He also agreed it’s probably time to revisit the dollar and casualty limits, which he said were borrowed from California law.

Cimino pushed back on one point: the draft already lets a court stop a lab’s internal use of a dangerous model, not just a public launch, and it requires a report if a model breaks out and reaches outside systems. CEO Dario Amodei publicly thanked Calvin for his scrutiny, saying the criticism came from a constructive, honest place.

The timing is awkward. Anthropic is preparing a stock market listing expected after the November midterms, and it has had a rocky year with Washington: the Trump administration banned it from Pentagon work in February, then forced it to suspend its most advanced models for nearly three weeks in June. Around the same time, Amodei urged the industry to slow down, warning that teams of AI agents might seize control of the whole internet inside a year.

For most people, nothing changes yet. The draft has no legal force, Congress hasn’t passed any federal AI law, and lawmakers were out of session until after the November elections. US AI policy under President Trump has leaned on voluntary pledges and executive orders, and Trump has brushed off calls to slow the technology down.

If something like this passed, though, you’d likely see more published safety testing for the biggest AI tools, and the companies behind them would have to report serious incidents. What you wouldn’t get is a regulator able to pull a chatbot for smaller harms like bad medical advice, privacy slips or scams that fall short of catastrophe. That trade-off, between predictable rules for labs and room for regulators to act, is the real fight.

Watch for Anthropic’s rewritten draft and whether any lawmaker formally introduces it. Separately, Senate Majority Leader John Thune and Sens. Amy Klobuchar and Ted Cruz have been negotiating their own bill on catastrophic AI risks, while Rep. Greg Casar and Sen. Bernie Sanders have called for halting advanced AI development altogether.