Meta AI Crawler Terms: What the Pause Means for Your Site

Meta has paused the business terms that would have let its Meta AI crawler collect content from any website tied to a Meta business account. On November 6, 2026, a Meta spokesperson told Reuters, “We’re pausing the changes as we continue to engage with stakeholders and take our time to get this right,” as reported by The Next Web and NewsBytes. Publishers are celebrating, but the pause leaves the main question open: who decides how bots use your site? Our answer is that you should decide, and you should write that decision down in places a bot actually reads, starting with robots.txt and your edge firewall.

This post covers what the terms said, why lawyers and regulators pushed back, which Meta crawlers hit WordPress sites today, and the crawl policy we think every business site should set this week.

What Meta’s Commercial Terms Said

Meta planned to update its Commercial Terms, which cover business accounts that manage Facebook pages and groups, run ads, send marketing messages or use the Business Suite dashboard. Nieman Lab first reported the change on October 15. The new language let Meta “access, index, cache, analyze, collect, retrieve, or crawl the domains, web pages, or URLs” connected to a business’s products or services. It also said Meta “may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train” its products.

That reach is wide. The Next Web notes the Commercial Terms apply to any business that uses Meta’s products, including one that only runs a Facebook Page. If your company has a Facebook Page that links to your WordPress site, your site was in scope. The terms offered no opt out. Businesses that disagreed were told to stop using the products for business purposes, per Nieman Lab.

The timeline so far

• October 15: Nieman Lab reports the new terms, set to take effect October 30.
• Mid October: Meta removes the word “train” and limits the purpose to “advertising and business products,” according to Search Engine Journal and Nieman Lab. The crawling access language stays the same.
• October 26: Meta moves the effective date to November 20, per Search Engine Journal and Nieman Lab.
• November 6: Meta pauses the changes. Its existing terms stay in effect, and it has not said when or how the crawling rules will return, according to The Next Web.

Why Publishers, Lawyers and Regulators Objected

The pushback came from several directions at once. Legal experts quoted by Nieman Lab read the original clause as clear permission to train AI on site content. James Grimmelmann, a Cornell Law professor, said the language plainly allowed that. Benjamin Sobel, a University of Wisconsin law professor, compared the setup to “tying” in antitrust law, because using Meta’s products meant accepting the crawling terms. Matt Topic, a media and IP attorney at Loevy & Loevy, said it was unclear whether a court would enforce the provision, but that publishers should assume it would be.

Regulators and courts moved too. According to The Next Web and NewsBytes:

• A Norwegian court temporarily blocked the terms after a lawsuit from Norway’s association of media companies.
• US senators Adam Schiff and Marsha Blackburn wrote to Mark Zuckerberg asking Meta to withdraw the terms.
• The UK Competition and Markets Authority said it would keep examining the terms. A CMA spokesperson told Nieman Lab, “We will continue to scrutinize these terms and engage with Meta and others as we consider our next steps.”
• The European Commission is assessing a publisher complaint, per The Next Web, and had earlier welcomed the postponement, per Nieman Lab.

Meta’s position has stayed consistent. A spokesperson told Search Engine Journal the crawling terms “only applied to our ads and business products” and that the update “does not change how Meta uses web crawlers.” Meta has also said site owners can still block its crawler with robots.txt, per Nieman Lab and The Next Web.

Theo Bamber, CEO of the UK News Media Association, called the October delay “a pause not a solution,” according to Nieman Lab. The same is true of the November pause.

The Robots.txt Problem

Here is the detail that matters most for site owners. Meta says robots.txt still works, but none of the versions of the terms mentioned robots.txt, according to Search Engine Journal and Nieman Lab. Legal experts told Nieman Lab that the terms could give Meta grounds to argue that a contract signed by the business overrides the blocking rules in its robots.txt file. Search Engine Journal also flagged that the clause let Meta “create test credentials” to crawl pages, and Grimmelmann told Nieman Lab that could technically let Meta sign up as a user to get past registration walls.

We are not lawyers, and nobody knows how a court would read these terms. What we do know is how bots behave. A crawler reads robots.txt and a firewall rule on every visit. It never reads the commercial terms you clicked through when you set up an ad account. If you want a bot policy that holds up, put it where the bot meets your site.

Which Meta AI Crawler Visits Your Site

Meta runs several bots, and they do different jobs. Blocking all of them would break things your marketing team relies on. Here is the list from Meta’s own crawler documentation, with details confirmed by PPC Land:

• facebookexternalhit: builds the title, description and thumbnail when someone shares your link on Facebook, Instagram or Messenger. Meta says it may skip robots.txt for security checks. Block it and your shared links lose their previews.
• Meta-ExternalAgent: crawls to train foundation AI models and to index content for Meta products. It follows robots.txt.
• Meta-WebIndexer: analyzes content to improve Meta AI search results. Meta says allowing it helps Meta cite and link to your content in Meta AI answers. It follows robots.txt.
• Meta-ExternalAds: supports advertising and other business products. It follows robots.txt.
• Meta-ExternalFetcher: fetches single links when a user asks, including to help AI complete tasks on websites. Meta says it may skip robots.txt because the fetch is user requested.

Note that Meta-ExternalAds lines up with the “advertising and business products” purpose in the revised terms. Each of these bots is a separate user agent, so a block on one does not block another.

Our Stance: Set Your Own Crawl Policy

The terms fight is between Meta, publishers and regulators, and it may run for months. Your site cannot wait for the outcome. We think every business site should decide, in writing, which bots get in. Our default for WordPress business sites is simple: block AI training crawlers, keep search crawlers and link preview bots, and make a deliberate choice about AI search bots.

The reasons are practical. Training crawlers collect content to build AI models. Search crawlers and link preview bots exist to send people to your pages. AI search bots sit in the middle: they can cite and link to you, and some businesses want that exposure. That third choice is a business call, and it should be yours.

There is also a server cost. Every uncached bot request uses PHP workers, database queries and bandwidth on your hosting plan. On a busy site, an aggressive crawler can slow pages for real customers. We covered how much a few hours of trouble costs a business in our Google Cloud outage post, and slow pages carry a similar cost on a smaller scale.

Step 1: Write the rules in robots.txt

WordPress serves a virtual robots.txt file unless a real one sits in your site root. Many SEO plugins let you edit it from wp-admin. To block Meta’s training crawler and keep everything else, add these lines:

User-agent: meta-externalagent
Disallow: /

If you also want to stay out of Meta AI search answers, add a second group for meta-webindexer. Leave facebookexternalhit alone so your shared posts keep their previews. Meta says robots.txt changes can take up to 24 hours to apply, because crawlers cache the file.

Keep a dated copy of each version of your robots.txt. If the legal question of whether a contract can override robots.txt ever matters to your business, a record of what you told each crawler, and when, is useful evidence.

Step 2: Enforce it at the edge

Robots.txt is a request. Bots that ignore it will still reach your server. An edge firewall turns the request into a rule. Cloudflare now lets every customer, including the Free plan, set separate policies for search, training and agent crawlers, according to Help Net Security and The Decoder. Both report that since September 15, 2026, Cloudflare blocks training and agent bots by default on pages with ads for new domains, while search crawlers stay allowed.

If your site already runs behind Cloudflare, open the AI crawler settings and confirm they match your robots.txt. A block in the firewall and an allow in robots.txt sends a mixed message, and so does the reverse.

Step 3: Watch the logs

Check your access logs or your Cloudflare analytics for the Meta user agents above. You want to see training crawlers getting blocked, preview fetches getting through, and no single bot eating a large share of your requests. If a bot you blocked keeps showing up with full page responses, your rule is not working.

What This Means for WordPress Business Sites

This story looks like a fight between big publishers and a big platform, but the terms reached any business with a Facebook Page. That includes local shops, agencies and online stores that never thought of themselves as publishers. The same pattern keeps showing up across the web: a third party changes its terms, and site owners find out after the fact. We saw a version of it with plugins in our Rank Math PRO backdoor post, where a trusted vendor channel became the risk.

The fix is the same in both cases. Control the layers you own. Your server, your edge network and your robots.txt file belong to you, and they apply to every bot no matter what terms you agreed to elsewhere. Our post on the WordPress core vulnerability made a similar point about security: the controls that hold are the ones at the web server and the edge.

At WP Farm, the sites we manage run behind Cloudflare, and our team can set AI crawler rules at the edge to match your robots.txt. If you want training bots blocked and search bots welcomed, tell us, and we will set it up and check the logs afterward.

Meta AI Crawler FAQ

Are Meta’s new crawling terms in effect?

No. Meta paused the changes on November 6, 2026, and its existing terms remain in place, according to The Next Web and NewsBytes. Meta has not said when the crawling rules will return or whether they will change.

Will blocking Meta-ExternalAgent hurt my Facebook link previews?

No. Link previews come from a different bot, facebookexternalhit. Meta-ExternalAgent is the crawler Meta lists for AI training. You can block the training crawler and keep previews working.

Does blocking AI training bots hurt my Google rankings?

Blocking Meta’s crawlers has no effect on Google. If you also block other AI bots, check each one’s purpose first, and keep search engine crawlers such as Googlebot and Bingbot allowed.

Is robots.txt legally binding?

That is the open question. Legal experts quoted by Nieman Lab said Meta’s terms could be used to argue a contract overrides robots.txt. We cannot predict how a court would rule, which is why we pair robots.txt with an edge firewall rule that bots cannot skip.

Take Control of Your Bot Traffic

You should decide who crawls your site, and your hosting should make that easy to enforce. See WP Farm managed WordPress hosting and get a team that sets your crawler rules at the edge and keeps an eye on the logs for you.