Anthropic’s safety bill would block AI releases only after $1B damage or 50 deaths

The Anthropic AI safety bill is a 37-page discussion draft, given the working name AEGIS Act (short for Artificial Intelligence Evaluation, Governance, and Incident Safety). Anthropic representatives shared it quietly with members of Congress in mid-October, and it soon circulated online. A company spokesperson said it reflects the Advanced AI Framework that Anthropic has promoted to federal and state lawmakers since June.

The bill only covers the biggest players. A company would count as a covered developer if it trains a model using more than 10²⁵ FLOPs (a measure of the raw computing power used in training) and also has either more than $500 million a year in AI revenue or over $1 billion a year in AI research spending.

Covered developers would have to:

• publish system cards explaining how their models work and how they were tested;
• adopt a written safety framework;
• report serious safety incidents;
• let independent evaluators examine their labs and models.

Those evaluators would register with the Commerce Department. Labs could pay them, as long as the money isn’t tied to what the evaluation finds. If an evaluator decides a model poses an imminent risk, it must tell Commerce within 72 hours of finishing its review.

The pause power is where the controversy sits. To block a model’s release or use, the government would have to show a foreseeable and imminent risk of more than $1 billion in damage, or of more than 50 people being killed or seriously injured. The harm also has to come from one of three places: a cyberattack, a biological weapon, or a model slipping out of its developer’s control.

That leaves out many everyday AI harms. AI-powered fraud, for example, would not qualify, no matter how many people it hit.

Even when the bar is met, the process is slow by design. Only the Commerce Department could act, and it would have to ask a federal judge at the District Court for the District of Columbia to order the suspension. The commerce secretary would need to sign off personally, the government would have to explain when the danger is expected and why other steps won’t work, and a temporary halt could last no more than 45 days. The draft says no other federal department, agency or official could stop a lab from developing, releasing or internally using a model.

The numbers themselves aren’t new. California’s SB 53, passed in 2025, uses the same 50-casualty and $1 billion definition of catastrophic risk. The difference is that California uses it to decide what companies must disclose and assess, while Anthropic’s draft uses it as the trigger for a court-ordered shutdown.

Nathan Calvin of Encode, an advocacy group that worked alongside Anthropic to pass California’s AI safety law, published a long critique asking what kind of company Anthropic really is. He argued the draft could make it harder for courts to stop dangerous models than it is under current law. California state senator Scott Wiener, who wrote SB 53, said on X that the plan falls well short of protecting public health and safety.

Critics compare it unfavorably with the FRONTIER Act, a bipartisan bill introduced in the House this summer by Reps. Jay Obernolte (R-Calif.) and Lori Trahan (D-Mass.). Charlie Bullock of the Institute for Law & AI called Anthropic’s effort “pretty reasonable” and more than other frontier labs have offered, but still weaker than the FRONTIER Act. His main worry: with no licensing standard for evaluators, auditors could compete by offering the cheapest, lightest checks.

States are the other flashpoint. The draft would override some state rules on disclosure, outside testing and incident reporting, while keeping state child safety and civil rights laws. Back in June, Anthropic had written that states should keep full authority to legislate until a strong federal law exists.

There’s also a fight over which models count. Some critics say the 10²⁵ FLOPs line is outdated: by June 2025, at least 33 models from 12 developers had already passed it. California’s law sets its compute trigger ten times higher.

Anthony Cimino, who leads federal affairs at Anthropic, admitted the state preemption wording was not precise enough and promised a rewrite. He said the dollar and casualty figures came from California law but agreed it may be time to revisit them. He also pointed out that the draft already lets a court stop a lab’s internal use of a model, not only a public launch.

CEO Dario Amodei then posted a reply of nearly 2,000 words on X. He thanked Calvin for the scrutiny, rejected his core argument, and named the draft bill as one of three things the company could have handled better, saying the responsibility was his. To show the company accepts real costs for safety, he said Anthropic held back wide access to its powerful Mythos model for about four months over cyberattack concerns, offering it first to a small group of cyber defense partners.

As of the latest reports, Anthropic had not published a revised draft or committed to lowering the 50-casualty bar.

For now, nothing changes. The AEGIS draft is not a bill anyone has formally introduced, and Congress has yet to pass any federal AI law. It was out of session until after the November midterms.

But the stakes are real. If a version of this becomes law, it would decide who can pull a harmful AI system off the market, how fast, and whether your state can set tougher rules than Washington. Other plans are competing for the same space, including Senate talks on catastrophic AI risk involving Majority Leader John Thune, Sen. Amy Klobuchar and Sen. Ted Cruz, and calls from Sen. Bernie Sanders to halt advanced AI development entirely.

The company also has a lot riding on its image. Its top models were shut off for nearly three weeks in June on federal orders, and it is preparing a stock market listing. How it rewrites this bill will be read as a test of whether its safety promises hold up when they cost something.