Cloudflare introduced Disallow AI Training, enabling site admins to block AI crawlers while keeping search engines. Apple, Google, and Microsoft have pledged compliance. The tool distinguishes accountable mixed crawlers from others, with automatic migration for most existing customers. Google is launching a transparency tool soon, Apple is developing a similar URL-level tool, and Microsoft plans to support robots.txt rejection for AI training by early 2027.





Home
