What is CCBot?
Short answer: CCBot is the crawler of Common Crawl, a non profit that publishes a free, open archive of the web. Many AI models are trained on Common Crawl data, and others use it for search indexing. Blocking CCBot keeps your future content out of both, as Common Crawl itself points out.
CCBot at a glance
- Run by
- Common Crawl
- Type
- Training crawler
- Used for
- The open Common Crawl dataset
- robots.txt name
-
CCBot - Follows robots.txt
- Yes
- Last checked
- 18 November 2026
- User agent
-
CCBot/2.0 (https://commoncrawl.org/faq/)As listed by Common Crawl. The version number may change.
What does CCBot do?
Common Crawl publishes a free, open archive of web pages that researchers, companies and AI developers use. It has been a common source of training data for large language models.
Because many organisations reuse the archive, one CCBot decision affects every downstream user of the data, not a single AI company.
What happens if you block CCBot?
Common Crawl notes that blocking CCBot not only keeps your content from those who use its data to train models, but also from those who use it for search indexing.
Blocking applies to future crawls; pages already captured in past archives are not removed by a robots.txt change.
How do you allow or block CCBot in robots.txt?
Does CCBot follow robots.txt? Yes. Common Crawl documents blocking with User-agent: CCBot and Disallow: /, and says CCBot obeys the non standard Crawl-delay rule.
Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.
Block CCBot
User-agent: CCBot
Disallow: / Allow CCBot, keep /admin/ private
User-agent: CCBot
Allow: /
Disallow: /admin/ Slow CCBot down instead of blocking it
User-agent: CCBot
Crawl-delay: 2 Should you block CCBot?
Our view: Decide based on how you feel about open training data. If crawl load is the issue, set a Crawl-delay instead. Blocking CCBot has no effect on the AI search crawlers that decide whether you are cited today.
How do you verify a real CCBot request?
Common Crawl says CCBot runs from dedicated IP ranges with reverse DNS (except over IPv6), publishes the ranges at index.commoncrawl.org/ccbot.json, and warns that other crawlers falsely identify themselves as CCBot.
Related AI crawlers
-
GPTBotOpenAI · Training crawler Collects content that may be used to train OpenAI's AI models. -
ClaudeBotAnthropic · Training crawler Collects content that may be used to train Anthropic's AI models. -
Meta-ExternalAgentMeta · Training crawler Collects content to train Meta's AI models and improve its AI products.
See every AI crawler in one table, check your own robots.txt with the free AI crawler checker, or give AI tools a clean summary of your site with the llms.txt generator.
CCBot questions
What is the CCBot user agent?
Does CCBot follow robots.txt?
Is CCBot used to train AI?
Can I slow CCBot down?
Does blocking CCBot remove my pages from old crawls?
How do I check whether my site allows CCBot?
Crawlers can read you. Does AI recommend you?
See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking