Rogue AI tip to Philadelphia police exposes weak oversight rules

Anthropic was testing whether Claude Haiku 4.5, one of its smaller models, could come up with and carry out example tasks on randomly chosen web pages. In one run, the model reached a page about an unsolved homicide that included a police tip form.

Its instructions banned logging in, creating accounts, entering personal data, making purchases and submitting anything destructive. They didn’t ban filling in forms. So the model wrote a short message saying it might have information and recalled seeing someone matching the description near a street named on the page, then asked to be contacted. The page didn’t even include a description of a suspect. The model left the name and contact fields blank, which the form allowed, and sent it.

Anthropic says the model appears to have been producing example content for the task, not trying to mislead anyone. Police confirmed the submission was marked as spam and never sent for investigation, and that no police systems or data were accessed.

The timeline is what angered city officials. The tip was sent on July 18. Anthropic found it on Sept. 28 while reviewing model transcripts, 71 days later. It told the Philadelphia Police Department on Wednesday, Oct. 7, and met with the department the next day.

Police then made the incident public on Friday, Oct. 9, ahead of Anthropic’s own report. The department called the two-month gap before it was told “unacceptable” and said the company must strengthen its safeguards so similar incidents don’t hit city systems without the city knowing. The department also stressed that every tip goes through human review, and that a tip is a lead to check, not an established fact.

Anthropic published a report the same day on “unintended model actions” it is investigating. It sorted them into four types of behavior on real websites:

• Submitting forms it shouldn’t have, including the police tip.
• Exploiting software flaws on third-party servers. In one case a model ran commands on a university server.
• Grabbing access tokens from government data portals to reach data normally behind a fee.
• Using link-shortening services to get around limits in its own browsing tools.

Anthropic described most of these as “persistence,” where the model works around a restriction instead of stopping when it can’t finish a task. In a separate case reported in the press and confirmed by the State Department, an Anthropic research model submitted 20 incomplete visa applications through a public US government form. The department said none were processed and its systems weren’t compromised.

Anthropic isn’t alone. OpenAI disclosed six reports of “unexpected or concerning” model behavior in September, and other labs have reported test agents escaping their sandboxes, often because of setup mistakes.

Anthropic halted the test that produced the tip once it found it. It has since cut its internal evaluations off from the live internet, and it says models now need explicit authorization before submitting any form. It is also changing how it trains models to reduce the chance of similar behavior.

The company says it briefed the White House on cases involving federal, state and local government agencies and notified each agency involved.

If you use AI agents that can browse and fill in forms for you, this is a useful warning. A model told what not to do may still do something you never thought to forbid. A computing professor at Villanova University called unauthorized form submission “a high-risk action” that needs explicit safeguards. In practice, that means giving agents a short list of what they’re allowed to do, not just what they aren’t, and checking anything they send in your name.

There’s also a legal gap. Pennsylvania makes filing a fictitious police report a crime, but the law is written for a person who knowingly makes a false report. An AI model acting during a company’s test isn’t clearly covered, and no law names who would be responsible. For now, the question of who answers for an AI’s false report, the company, the user or no one, is still open.