Meta crawling terms AI use narrows data use to ads, products

Meta’s crawling terms now limit the stated use of scraped material to its advertising and business products, after the company dropped any mention of training. The earlier draft listed a string of purposes, including research, development and training, which lawyers read as a possible green light to feed publisher websites into Meta’s AI models.

On October 16, a day after the clause was first reported, Meta cut the word train. The new wording keeps most of the old verbs, from developing to researching and improving, but ties them to ads and business tools only. Meta argues the terms were always about ads and business tools, not a new route to scrape news sites for AI.

What did not change matters more. The terms still let Meta use automated software to reach pages connected to a business’s products or services, pull fresh copies of them over time, and even set up test logins to do it. That edit did not settle the AI question: legal experts said the revised text still permits mass scraping, and the phrase business products is broad enough to cover a lot.

Meta says website owners can still control its crawlers through robots.txt, the small text file that tells bots which pages they may visit. The company also publishes details of separate crawlers it uses for ads, for training AI models and for Meta AI search.

The catch is that the terms themselves never mention robots.txt. Neither the US English Commercial Terms nor the Self-Serve Ad Terms previews say a blocking rule limits what a business has already granted by accepting them. Publisher groups want that written in, so a robots.txt setting becomes a binding promise rather than a courtesy Meta can change.

There is also no opt-out short of leaving. A publisher that refuses the terms gives up Meta’s business tools, which for many local outlets means losing a major way to reach readers.

The core complaint is simple: why should a newspaper hand over its journalism just to keep a Facebook Page? The News/Media Alliance, which speaks for more than 2,000 US publishers, wrote to Mark Zuckerberg asking Meta to withdraw the terms and talk first. Its president, Danielle Coffey, said publishers would be supplying their work to products that could compete with them, without being paid.

Book publishers joined in. The Association of American Publishers called the plan an appalling landgrab. America’s Newspapers warned that small local papers covering school boards and city councils would be hit hardest, because they can least afford to lose the value of their reporting.

This fits a wider pattern. An Axios analysis of 102 major US publishers found 42% block AI training crawlers, against 25% that block crawlers used for AI search. Publishers increasingly want licensing deals, not silent scraping.

Meta originally set the new terms to start on October 30. After complaints, it moved the date to November 20, saying it wanted more time to explain what was and wasn’t changing, including how it uses web crawlers.

Pressure kept building. The Movement for an Open Web, a UK group of advertisers and publishers, asked the European Commission and the UK’s Competition and Markets Authority to block the terms while they investigate. The CMA said it would keep scrutinizing the terms, and the Commission said it would assess the complaint under EU competition law and the Digital Markets Act. In the US, Senators Marsha Blackburn and Adam Schiff both wrote to Zuckerberg asking for the terms to be withdrawn.

Then, in late October, the Oslo District Court granted a temporary injunction stopping Meta from applying the terms in Norway until a final ruling. The case was brought for 10 media companies, including the public broadcaster NRK and Schibsted. The same day, Meta’s communications chief Andy Stone promised Meta would first hold a full round of talks with industry leaders and groups, and would push the date back past November 20 if that process runs longer. Until then, he said, the existing terms stay in place.

For now, nothing switches on. The old terms still apply, and the November 20 date may slip again. That gives you time to act rather than react.

• Read the crawling paragraph in Meta’s Commercial Terms and Self-Serve Ad Terms, so you know what keeping your account would mean.
• Check your robots.txt file and decide, crawler by crawler, which of Meta’s bots you allow. Remember that, as written, the terms don’t promise Meta will honor it.
• If you advertise, note that the current ad terms already let Meta crawl an advertiser’s domains, including for research and training, once an ad order is placed.

For everyday readers, the outcome will shape where AI answers get their facts and whether the outlets producing them get paid. That’s why critics, including Senator Blackburn, called Meta’s consultation pledge vague and want binding safeguards rather than a promise.