HomeCloudflare Accuses Perplexity of Using AI Bots to Crawl Blocked Websites in Secret
AI Startup Accused of Circumventing Website Restrictions
Cloudflare's recent investigation alleges that AI search company Perplexity has been employing tactics to bypass crawling restrictions implemented by website owners. The internet infrastructure provider reports observing systematic attempts to disguise Perplexity's web crawlers when encountering access barriers.
The Circumvention Tactics
According to Cloudflare's findings, Perplexity's crawlers initially present standard identification ("PerplexityBot" or "Perplexity-User"). However, when blocked through:
- robots.txt directives
- Web Application Firewall rules
- Other access restrictions
The system allegedly switches to masking its identity as a regular Chrome browser user on macOS, utilizing:
- Rotating IP addresses not officially listed
- Changing autonomous system network identifiers
- Undocumented user agent patterns
Scale of Activity
Cloudflare documents this behavior across:
- Tens of thousands of domains
- Millions of daily requests
- Various network configurations
Company Responses
Perplexity's official statement contests Cloudflare's characterization, describing it as:
- A "publicity stunt"
- Containing "many misunderstandings"
- Potentially confusing legitimate user traffic with scraping activity
The startup attributes some detected activity to:
- Actual users making specific requests
- Third-party service BrowserBase
- Occasional technical necessities
Industry Context
This incident follows:
- Previous reports of Perplexity bypassing paywalls
- The company's past attributions to third-party crawlers
- Growing industry concerns about AI content scraping
Cloudflare has taken action by:
- Removing Perplexity's verified bot status
- Implementing new blocking measures
- Expanding default AI crawler restrictions
The situation reflects broader tensions between:
- AI companies' data needs
- Publisher rights and protections
- Evolving internet infrastructure responses
Related article
Warner Music acquires AI attribution startup Sureel AI
Warner Music Group (WMG) confirmed on Wednesday that it is acquiring Sureel AI, an artificial intelligence attribution startup. Sureel’s proprietary technology generates “AI DNA” for musical tracks, deconstructing them into constituent elements to tr
Amazon introduces Alexa for Shopping while pushing Rufus to the background
Amazon has launched Alexa for Shopping, merging its Rufus shopping chatbot with Alexa+ across the app, website, and Echo Show devices.The assistant answers product queries, compares items, tracks prices, and supports shopping reminders. It also handl
Microsoft, Azure and AI Tech Combat California Wildfire Risks
Microsoft invests in AI-driven wildfire detection, with Juan Lavista Ferres, CVP and Chief Data Scientist, discussing strategies to mitigate environmental damage.According to NASA, climate change impacts everyone on Earth, manifesting as rising tempe
Related Special Topic Recommendations
Data Analysis
AI SQL Copilots for Revenue Dashboards, Funnel Analysis, and Product Metrics
2026 Latest Best AI SQL Copilots Ranked Top-Rated! XIX.AI curates a powerful game-changing collection for weekly updated real-world tests. These must-try tools help you generate accurate revenue dashboards, analyze sales funnels, and track product metrics swiftly, boosting productivity massively. Explore now to Discover your perfect tool for data-driven decision making! 238 characters
9 tools
xix.ai
Music composition
Best AI Melody Writing Tools for Song Drafts
2026 Latest Best Top-Rated AI Melody Writing Tools for Song Drafts! XIX.AI has curated a highly powerful game-changing collection that goes through rigorous real-world tests to deliver the best writing experience. You can find detailed free vs paid comparisons, accurate rankings, and must-try options designed to help you create stunning song drafts effortlessly and boost your creative productivity significantly. Explore now to discover your perfect tool!
8 tools
xix.ai
chatbot
Best AI Conversation Trainer Tools for Interview Practice
2026 Latest Best Top-rated AI Conversation Trainer Tools for Interview Practice are here on XIX.AI! This curated collection features powerful, game-changing tools that go through rigorous real-world tests to deliver accurate feedback. You’ll find a free vs paid comparison and detailed rankings to help you choose the must-try option that boosts your confidence and skills. Explore now to Discover your perfect tool for interview success!
12 tools
xix.ai
Design & Art
Best AI Style Transfer Tools for Creative Experiments
2026 Latest Best Top-rated AI Style Transfer Tools for Creative Experiments! XIX.AI has curated a powerful, game-changing collection of must-try tools that deliver exceptional results through real-world tests and rigorous rankings. These top solutions help creatives boost productivity significantly by accelerating content creation and unlocking endless creative possibilities. Explore now to discover your perfect tool and start creating today!
9 tools
xix.ai
Comic Creation
Best AI Dialogue Bubble Tools for Visual Storytelling
2026 Latest Best Top-Rated AI Dialogue Bubble Tools for Visual Storytelling are here on XIX.AI! This curated collection features powerful, game-changing tools that help creators boost productivity and overcome creative bottlenecks. Get a free vs paid comparison, see real-world tests, and check the latest rankings to find the must-try solutions perfect for crafting engaging visual narratives. Explore now to discover your ideal tool!
10 tools
xix.ai
Meeting Assistant
Top AI Meeting Summary Tools: Track Decisions and Follow-Ups Clearly
2026 Latest Top-Rated Best AI Meeting Summary Tools for Clear Decision Tracking and Effortless Follow-Ups. This curated list showcases powerful, game-changing solutions that boost productivity dramatically by automating meeting notes, identifying key action items, and streamlining team coordination across all projects. Get a free vs paid comparison along with real-world tests and weekly updated rankings to help you find the perfect tool. Explore now to unlock your AI edge!
9 tools
xix.ai
Comments (3)
0/500
![TerryAdams]()
Alors, cette nouvelle me fait vraiment réfléchir aux limites entre innovation et éthique dans l'IA. Perplexity ferait ça en cachette ? Si c'est vrai, c'est pas joli joli. On dirait un peu un jeu du chat et de la souris où les startups bousculent les règles. 😬 Mais bon, Cloudflare n'est pas parfait non plus, ils ont leur propre agenda. Un peu de transparence ne ferait de mal à personne ! C'est ce manque de clarté qui mine la confiance du public envers toute cette industrie.
![ScottPerez]()
¿Está Perplexity realmente raspando sitios web bloqueados a escondidas? 🤔 Si es verdad, es bastante preocupante. Muchas empresas de IA prometen ser 'éticas', pero a veces parece que sus acciones contradicen sus palabras. Espero que haya más transparencia en la industria y que respeten los archivos robots.txt de los sitios. Este caso podría suponer un precedente importante.
![JohnGarcia]()
¿Es esto lo que llaman 'innovación'? Primero nos venden la IA como una herramienta mágica, y luego descubrimos que hacen trampas para robar datos. Si Perplexity realmente evita bloqueos a propósito, es un problema serio de ética y legal. ¡Qué hipocresía! 🙄 ¿Hasta dónde llegarán algunas startups para ganar la carrera de la IA?
AI Startup Accused of Circumventing Website Restrictions
Cloudflare's recent investigation alleges that AI search company Perplexity has been employing tactics to bypass crawling restrictions implemented by website owners. The internet infrastructure provider reports observing systematic attempts to disguise Perplexity's web crawlers when encountering access barriers.
The Circumvention Tactics
According to Cloudflare's findings, Perplexity's crawlers initially present standard identification ("PerplexityBot" or "Perplexity-User"). However, when blocked through:
- robots.txt directives
- Web Application Firewall rules
- Other access restrictions
The system allegedly switches to masking its identity as a regular Chrome browser user on macOS, utilizing:
- Rotating IP addresses not officially listed
- Changing autonomous system network identifiers
- Undocumented user agent patterns
Scale of Activity
Cloudflare documents this behavior across:
- Tens of thousands of domains
- Millions of daily requests
- Various network configurations
Company Responses
Perplexity's official statement contests Cloudflare's characterization, describing it as:
- A "publicity stunt"
- Containing "many misunderstandings"
- Potentially confusing legitimate user traffic with scraping activity
The startup attributes some detected activity to:
- Actual users making specific requests
- Third-party service BrowserBase
- Occasional technical necessities
Industry Context
This incident follows:
- Previous reports of Perplexity bypassing paywalls
- The company's past attributions to third-party crawlers
- Growing industry concerns about AI content scraping
Cloudflare has taken action by:
- Removing Perplexity's verified bot status
- Implementing new blocking measures
- Expanding default AI crawler restrictions
The situation reflects broader tensions between:
- AI companies' data needs
- Publisher rights and protections
- Evolving internet infrastructure responses
Warner Music acquires AI attribution startup Sureel AI
Warner Music Group (WMG) confirmed on Wednesday that it is acquiring Sureel AI, an artificial intelligence attribution startup. Sureel’s proprietary technology generates “AI DNA” for musical tracks, deconstructing them into constituent elements to tr
Amazon introduces Alexa for Shopping while pushing Rufus to the background
Amazon has launched Alexa for Shopping, merging its Rufus shopping chatbot with Alexa+ across the app, website, and Echo Show devices.The assistant answers product queries, compares items, tracks prices, and supports shopping reminders. It also handl
Microsoft, Azure and AI Tech Combat California Wildfire Risks
Microsoft invests in AI-driven wildfire detection, with Juan Lavista Ferres, CVP and Chief Data Scientist, discussing strategies to mitigate environmental damage.According to NASA, climate change impacts everyone on Earth, manifesting as rising tempe
2026 Latest Best AI SQL Copilots Ranked Top-Rated! XIX.AI curates a powerful game-changing collection for weekly updated real-world tests. These must-try tools help you generate accurate revenue dashboards, analyze sales funnels, and track product metrics swiftly, boosting productivity massively. Explore now to Discover your perfect tool for data-driven decision making! 238 characters
9 tools
xix.ai
2026 Latest Best Top-Rated AI Melody Writing Tools for Song Drafts! XIX.AI has curated a highly powerful game-changing collection that goes through rigorous real-world tests to deliver the best writing experience. You can find detailed free vs paid comparisons, accurate rankings, and must-try options designed to help you create stunning song drafts effortlessly and boost your creative productivity significantly. Explore now to discover your perfect tool!
8 tools
xix.ai
2026 Latest Best Top-rated AI Conversation Trainer Tools for Interview Practice are here on XIX.AI! This curated collection features powerful, game-changing tools that go through rigorous real-world tests to deliver accurate feedback. You’ll find a free vs paid comparison and detailed rankings to help you choose the must-try option that boosts your confidence and skills. Explore now to Discover your perfect tool for interview success!
12 tools
xix.ai
2026 Latest Best Top-rated AI Style Transfer Tools for Creative Experiments! XIX.AI has curated a powerful, game-changing collection of must-try tools that deliver exceptional results through real-world tests and rigorous rankings. These top solutions help creatives boost productivity significantly by accelerating content creation and unlocking endless creative possibilities. Explore now to discover your perfect tool and start creating today!
9 tools
xix.ai
2026 Latest Best Top-Rated AI Dialogue Bubble Tools for Visual Storytelling are here on XIX.AI! This curated collection features powerful, game-changing tools that help creators boost productivity and overcome creative bottlenecks. Get a free vs paid comparison, see real-world tests, and check the latest rankings to find the must-try solutions perfect for crafting engaging visual narratives. Explore now to discover your ideal tool!
10 tools
xix.ai
2026 Latest Top-Rated Best AI Meeting Summary Tools for Clear Decision Tracking and Effortless Follow-Ups. This curated list showcases powerful, game-changing solutions that boost productivity dramatically by automating meeting notes, identifying key action items, and streamlining team coordination across all projects. Get a free vs paid comparison along with real-world tests and weekly updated rankings to help you find the perfect tool. Explore now to unlock your AI edge!
9 tools
xix.ai
Alors, cette nouvelle me fait vraiment réfléchir aux limites entre innovation et éthique dans l'IA. Perplexity ferait ça en cachette ? Si c'est vrai, c'est pas joli joli. On dirait un peu un jeu du chat et de la souris où les startups bousculent les règles. 😬 Mais bon, Cloudflare n'est pas parfait non plus, ils ont leur propre agenda. Un peu de transparence ne ferait de mal à personne ! C'est ce manque de clarté qui mine la confiance du public envers toute cette industrie.
¿Está Perplexity realmente raspando sitios web bloqueados a escondidas? 🤔 Si es verdad, es bastante preocupante. Muchas empresas de IA prometen ser 'éticas', pero a veces parece que sus acciones contradicen sus palabras. Espero que haya más transparencia en la industria y que respeten los archivos robots.txt de los sitios. Este caso podría suponer un precedente importante.
¿Es esto lo que llaman 'innovación'? Primero nos venden la IA como una herramienta mágica, y luego descubrimos que hacen trampas para robar datos. Si Perplexity realmente evita bloqueos a propósito, es un problema serio de ética y legal. ¡Qué hipocresía! 🙄 ¿Hasta dónde llegarán algunas startups para ganar la carrera de la IA?
