Skip to content
Training crawler Run by Common Crawl

What is CCBot?

Short answer: CCBot is the crawler of Common Crawl, a non profit that publishes a free, open archive of the web. Many AI models are trained on Common Crawl data, and others use it for search indexing. Blocking CCBot keeps your future content out of both, as Common Crawl itself points out.

CCBot at a glance

Run by
Common Crawl
Type
Training crawler
Used for
The open Common Crawl dataset
robots.txt name
CCBot
Follows robots.txt
Yes
Last checked
18 November 2026
User agent
CCBot/2.0 (https://commoncrawl.org/faq/) As listed by Common Crawl. The version number may change.

What does CCBot do?

Common Crawl publishes a free, open archive of web pages that researchers, companies and AI developers use. It has been a common source of training data for large language models.

Because many organisations reuse the archive, one CCBot decision affects every downstream user of the data, not a single AI company.

What happens if you block CCBot?

Common Crawl notes that blocking CCBot not only keeps your content from those who use its data to train models, but also from those who use it for search indexing.

Blocking applies to future crawls; pages already captured in past archives are not removed by a robots.txt change.

How do you allow or block CCBot in robots.txt?

Does CCBot follow robots.txt? Yes. Common Crawl documents blocking with User-agent: CCBot and Disallow: /, and says CCBot obeys the non standard Crawl-delay rule.

Add one of these groups to the robots.txt file at the root of your site. Several User-agent lines in a row share the rules below them, and a crawler follows only the most specific group that names it, so repeat any private paths in that group.

Block CCBot

User-agent: CCBot
Disallow: /

Allow CCBot, keep /admin/ private

User-agent: CCBot
Allow: /
Disallow: /admin/

Slow CCBot down instead of blocking it

User-agent: CCBot
Crawl-delay: 2

Should you block CCBot?

Our view: Decide based on how you feel about open training data. If crawl load is the issue, set a Crawl-delay instead. Blocking CCBot has no effect on the AI search crawlers that decide whether you are cited today.

How do you verify a real CCBot request?

Common Crawl says CCBot runs from dedicated IP ranges with reverse DNS (except over IPv6), publishes the ranges at index.commoncrawl.org/ccbot.json, and warns that other crawlers falsely identify themselves as CCBot.

CCBot questions

What is the CCBot user agent?
Common Crawl lists it as: CCBot/2.0 (https://commoncrawl.org/faq/). As listed by Common Crawl. The version number may change.
Does CCBot follow robots.txt?
Yes. Common Crawl documents blocking with User-agent: CCBot and Disallow: /, and says CCBot obeys the non standard Crawl-delay rule.
Is CCBot used to train AI?
Common Crawl data is widely used to train AI models, and also for research and search indexing. Common Crawl itself publishes the data rather than training models.
Can I slow CCBot down?
Yes. Common Crawl says CCBot obeys Crawl-delay; for example Crawl-delay: 2 asks for no more than one request every two seconds.
Does blocking CCBot remove my pages from old crawls?
No. A robots.txt rule affects future crawling; it does not remove pages from archives that were already published.
How do I check whether my site allows CCBot?
Run your domain through the free ONEGO AI crawler checker. It reads your robots.txt and shows whether CCBot and 15 other AI crawlers are allowed, blocked or partly restricted, and what each rule means.

Crawlers can read you. Does AI recommend you?

See your ONE Score first · 7 day free trial, no card needed · Pay only if you keep tracking