What is Meta-ExternalAgent?
Short answer: Meta-ExternalAgent is Meta's AI crawler. Meta says it crawls the web for uses such as training foundation AI models or improving products by indexing content directly. You can block it with a robots.txt disallow, but note that its sibling Meta-ExternalFetcher, which fetches pages users ask for, may bypass robots.txt.
Meta-ExternalAgent at a glance
- Run by
- Meta
- Type
- Training crawler
- Used for
- Meta AI training and products
- robots.txt name
-
Meta-ExternalAgent - Follows robots.txt
- Yes
- Last checked
- 18 November 2026
- User agent
-
meta-externalagentMeta's documentation writes the token in lower case. robots.txt matching is not case sensitive.
What does Meta-ExternalAgent do?
Meta says Meta-ExternalAgent crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. That covers both training and product uses under one token.
Meta runs a separate agent, Meta-ExternalFetcher, for fetches a user requests, for example when an AI agent visits a page to complete a task. Meta says it may bypass robots.txt because the fetch was requested by a user.
What happens if you block Meta-ExternalAgent?
Blocking Meta-ExternalAgent keeps your content out of Meta's AI training and out of the product indexing it uses that crawler for. Meta says robots.txt may be cached for up to 24 hours, so changes are not instant.
It does not control Meta-ExternalFetcher, and it does not affect link previews on Facebook and Instagram, which use Meta's separate sharing crawler.
How do you allow or block Meta-ExternalAgent in robots.txt?
Does Meta-ExternalAgent follow robots.txt? Yes. Meta says to block its crawlers by adding a disallow for the relevant crawler to robots.txt, and that crawlers may cache robots.txt for up to 24 hours.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Block Meta-ExternalAgent
User-agent: Meta-ExternalAgent
Disallow: / Allow Meta-ExternalAgent, keep /admin/ private
User-agent: Meta-ExternalAgent
Allow: /
Disallow: /admin/ Should you block Meta-ExternalAgent?
Our view: A business choice. Because one token covers both training and product indexing, blocking it may also reduce how Meta's AI products can use your pages. Leave it allowed if you want Meta AI to know your brand.
How do you verify a real Meta-ExternalAgent request?
Meta's web crawlers page is the reference for its agents. Treat the user agent alone as easy to fake.
Related AI crawlers
-
GPTBotOpenAI · Training crawler Collects content that may be used to train OpenAI's AI models. -
ClaudeBotAnthropic · Training crawler Collects content that may be used to train Anthropic's AI models. -
CCBotCommon Crawl · Training crawler Builds the open Common Crawl dataset, which many AI models are trained on.
See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
Meta-ExternalAgent questions
What is the Meta-ExternalAgent user agent?
Does Meta-ExternalAgent follow robots.txt?
What is the difference between Meta-ExternalAgent and Meta-ExternalFetcher?
How quickly does a Meta-ExternalAgent block take effect?
Does blocking Meta-ExternalAgent stop Facebook link previews?
How do I check whether my site allows Meta-ExternalAgent?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking