Cloudflare Accuses Perplexity of Using AI Bots to Crawl Blocked Websites in Secret

AI Startup Accused of Circumventing Website Restrictions

Cloudflare's recent investigation alleges that AI search company Perplexity has been employing tactics to bypass crawling restrictions implemented by website owners. The internet infrastructure provider reports observing systematic attempts to disguise Perplexity's web crawlers when encountering access barriers.

The Circumvention Tactics

According to Cloudflare's findings, Perplexity's crawlers initially present standard identification ("PerplexityBot" or "Perplexity-User"). However, when blocked through:

• robots.txt directives
• Web Application Firewall rules
• Other access restrictions

The system allegedly switches to masking its identity as a regular Chrome browser user on macOS, utilizing:

• Rotating IP addresses not officially listed
• Changing autonomous system network identifiers
• Undocumented user agent patterns

Scale of Activity

Cloudflare documents this behavior across:

• Tens of thousands of domains
• Millions of daily requests
• Various network configurations

Company Responses

Perplexity's official statement contests Cloudflare's characterization, describing it as:

• A "publicity stunt"
• Containing "many misunderstandings"
• Potentially confusing legitimate user traffic with scraping activity

The startup attributes some detected activity to:

• Actual users making specific requests
• Third-party service BrowserBase
• Occasional technical necessities

Industry Context

This incident follows:

• Previous reports of Perplexity bypassing paywalls
• The company's past attributions to third-party crawlers
• Growing industry concerns about AI content scraping

Cloudflare has taken action by:

• Removing Perplexity's verified bot status
• Implementing new blocking measures
• Expanding default AI crawler restrictions

The situation reflects broader tensions between:

• AI companies' data needs
• Publisher rights and protections
• Evolving internet infrastructure responses