Skip to content
Training crawler Run by Meta

What is Meta-ExternalAgent?

Short answer: Meta-ExternalAgent is Meta's AI crawler. Meta says it crawls the web for uses such as training foundation AI models or improving products by indexing content directly. You can block it with a robots.txt disallow, but note that its sibling Meta-ExternalFetcher, which fetches pages users ask for, may bypass robots.txt.

Meta-ExternalAgent at a glance

Run by
Meta
Type
Training crawler
Used for
Meta AI training and products
robots.txt name
Meta-ExternalAgent
Follows robots.txt
Yes
Last checked
18 November 2026
User agent
meta-externalagent Meta's documentation writes the token in lower case. robots.txt matching is not case sensitive.

What does Meta-ExternalAgent do?

Meta says Meta-ExternalAgent crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. That covers both training and product uses under one token.

Meta runs a separate agent, Meta-ExternalFetcher, for fetches a user requests, for example when an AI agent visits a page to complete a task. Meta says it may bypass robots.txt because the fetch was requested by a user.

What happens if you block Meta-ExternalAgent?

Blocking Meta-ExternalAgent keeps your content out of Meta's AI training and out of the product indexing it uses that crawler for. Meta says robots.txt may be cached for up to 24 hours, so changes are not instant.

It does not control Meta-ExternalFetcher, and it does not affect link previews on Facebook and Instagram, which use Meta's separate sharing crawler.

How do you allow or block Meta-ExternalAgent in robots.txt?

Does Meta-ExternalAgent follow robots.txt? Yes. Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and that crawlers may cache robots.txt for up to 24 hours.

Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.

Block Meta-ExternalAgent

User-agent: Meta-ExternalAgent
Disallow: /

Allow Meta-ExternalAgent, keep /admin/ private

User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/

Should you block Meta-ExternalAgent?

Our view: A business choice. Because one token covers both training and product indexing, blocking it may also reduce how Meta's AI products can use your pages. Leave it allowed if you want Meta AI to know your brand.

How do you verify a real Meta-ExternalAgent request?

Meta's web crawlers page is the reference for its agents. Treat the user agent alone as easy to fake.

Meta-ExternalAgent questions

What is the Meta-ExternalAgent user agent?
Requests contain meta-externalagent in the user agent. Meta's documentation writes the token in lower case. robots.txt matching is not case sensitive.
Does Meta-ExternalAgent follow robots.txt?
Yes. Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and that crawlers may cache robots.txt for up to 24 hours.
What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher?
Meta-ExternalAgent crawls for AI training and product indexing; Meta-ExternalFetcher fetches pages a user requested and, Meta says, may bypass robots.txt.
How quickly does a Meta-ExternalAgent block take effect?
Meta says its crawlers may cache robots.txt for up to 24 hours.
Does blocking Meta-ExternalAgent stop Facebook link previews?
No. Link previews use Meta's separate sharing crawler, not Meta-ExternalAgent.
How do I check whether my site allows Meta-ExternalAgent?
Run your domain through the free ONEGO AI crawler checker. It reads your robots.txt and shows whether Meta-ExternalAgent and 15 other AI crawlers are allowed, blocked or partly restricted, and what each rule means.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking