Cloudflare expands robots.txt with new AI content signals, giving publishers control over search, AI input, and AI training.

Internet infrastructure firm Cloudflare is launching a suite of tools that could help shift the power dynamic between AI companies and the websites they crawl for data. Today it’s giving all of its ...

UPDATE: Jun. 12, 2025, 5:06 PM EDT According to the Cloudflare status page, the service issues have been fully resolved. "All Cloudflare services have been restored and are now fully operational," the ...

Students expect seamless digital experiences, but rising cyber threats and outdated systems make that harder to deliver.

New Content Signals Policy will empower website owners to declare preferences on how AI companies access and use their ...

Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites. In a blog Tuesday, Cloudflare explained that its "pay-per-crawl" ...

SAN FRANCISCO--(BUSINESS WIRE)--Cloudflare, Inc. (NYSE: NET), the leading connectivity cloud company, today unveiled Cloudflare for AI, a suite of tools to provide comprehensive visibility, security ...

Update 7:57 p.m. ET: The issue has been solved, and Cloudflare’s status page says that all systems are operational. If you’ve noticed some internet slowdowns or trouble reaching websites tonight, ...

Cloudflare accused AI answer engine Perplexity of “stealth crawling,” saying it uses deceptive techniques to bypass website blocks and access content it’s been explicitly told not to touch. The big ...

AI startup Perplexity is crawling and scraping content from websites that have explicitly indicated they don’t want to be scraped, according to internet infrastructure provider Cloudflare. On Monday, ...