Changelog

Subscribe to RSS

2025-10-21

New Robots.txt tab for tracking crawler compliance

AI Crawl Control now includes a Robots.txt tab that provides insights into how AI crawlers interact with your robots.txt files.

What's new

The Robots.txt tab allows you to:

• Monitor the health status of robots.txt files across all your hostnames, including HTTP status codes, and identify hostnames that need a robots.txt file.
• Track the total number of requests to each robots.txt file, with breakdowns of successful versus unsuccessful requests.
• Check whether your robots.txt files contain Content Signals ↗ directives for AI training, search, and AI input.
• Identify crawlers that request paths explicitly disallowed by your robots.txt directives, including the crawler name, operator, violated path, specific directive, and violation count.
• Filter robots.txt request data by crawler, operator, category, and custom time ranges.

Take action

When you identify non-compliant crawlers, you can:

• Block the crawler in the Crawlers tab
• Create custom WAF rules for path-specific security
• Use Redirect Rules to guide crawlers to appropriate areas of your site

To get started, go to AI Crawl Control > Robots.txt in the Cloudflare dashboard. Learn more in the Track robots.txt documentation.

2025-10-14

Enhanced AI Crawl Control metrics with new drilldowns and filters

AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites.

What's new

Track crawler requests over time

Visualize crawler activity patterns over time, and group data by different dimensions:

• By Crawler — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider)
• By Category — Analyze crawler purpose or type
• By Operator — Discover which companies (OpenAI, Anthropic, ByteDance) are crawling your site
• By Host — Break down activity across multiple subdomains
• By Status Code — Monitor HTTP response codes to crawlers (200s, 300s, 400s, 500s)

Interactive chart showing crawler requests over time with filterable dimensions

Analyze referrer data (Paid plans)

Identify traffic sources with referrer analytics:

• View top referrers driving traffic to your site
• Understand discovery patterns and content popularity from AI operators

Bar chart showing top referrers and their respective traffic volumes

Export data

Download your filtered view as a CSV:

• Includes all applied filters and groupings
• Useful for custom reporting and deeper analysis

Get started

1. Log in to the Cloudflare dashboard, and select your account and domain.
2. Go to AI Crawl Control > Metrics.
3. Use the grouping tabs to explore different views of your data.
4. Apply filters to focus on specific crawlers, time ranges, or response codes.
5. Select Download CSV to export your filtered data for further analysis.

Learn more about AI Crawl Control.

2025-08-27

Enhanced crawler insights and custom 402 responses

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available.

Enhanced Crawlers tab:

• View total allowed and blocked requests for each AI crawler
• Trend charts show crawler activity over your selected time range per crawler

Custom block responses (paid plans): You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms.

For users on paid plans, when blocking AI crawlers you can configure:

• Response code: Choose between 403 Forbidden or 402 Payment Required
• Response body: Add a custom message with your licensing contact information

Example 402 response:

HTTP 402 Payment Required
Date: Mon, 24 Aug 2025 12:56:49 GMT
Content-type: application/json
Server: cloudflare
Cf-Ray: 967e8da599d0c3fa-EWR
Cf-Team: 2902f6db750000c3fa1e2ef400000001

{
  "message": "Please contact the site owner for access."
}

2025-07-01

Introducing Pay Per Crawl (private beta)

We are introducing a new feature of AI Crawl Control — Pay Per Crawl. Pay Per Crawl enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.

For Site Owners:

• Set pricing and select which crawlers to charge for content access
• Manage payments via Stripe
• Monitor analytics on successful content deliveries

For AI Crawler Owners:

• Use HTTP headers to request and accept pricing
• Receive clear confirmations on charges for accessed content

Learn more in the Pay Per Crawl documentation.

2025-07-01

AI Crawl Control refresh

We redesigned the AI Crawl Control dashboard to provide more intuitive and granular control over AI crawlers.

• From the new AI Crawlers tab: block specific AI crawlers.
• From the new Metrics tab: view AI Crawl Control metrics.

To get started, explore:

• Manage AI crawlers.
• Analyze AI traffic.

2024-09-23

AI Crawl Control

Every site on Cloudflare now has access to AI Audit, which summarizes the crawling behavior of popular and known AI services.

You can use this data to:

• Understand how and how often crawlers access your site (and which content is the most popular).
• Block specific AI bots accessing your site.
• Use Cloudflare to enforce your robots.txt policy via an automatic WAF rule.

To get started, explore AI audit.