Red Hot Cyber
Cybersecurity is about sharing. Recognize the risk, combat it, share your experiences, and encourage others to do better than you.
Redazione RHC : 5 December 2025 16:21
Since July 1st, Cloudflare has blocked 416 billion requests from artificial intelligence bots attempting to extract content from its customers’ websites. According to Matthew Prince, co-founder and CEO of Cloudflare, efforts to protect resources from unwanted crawling are already having a noticeable impact and could transform the traditional internet economy.
The impetus for these measures was provided by the Content Independence Day initiative, launched by Cloudflare this summer in collaboration with major publishers and AI companies. Its essence is to block AI crawlers’ access to content by default, unless the model’s developers have entered into a separate paid agreement.
Since July 2024, Cloudflare has been offering customers ready-to-use tools to automatically filter traffic from bots that collect data for AI training. According to the company, 416 billion such requests have already been blocked since July 1, 2025.
Prince argues that for decades, the internet’s business model has been based on a clear logic: creators publish content, generate traffic, and then monetize it through advertising, sales, or subscriptions. The advent of generative artificial intelligence, he argues, has revolutionized the platform: models now collect and process other people’s content, and users increasingly receive responses directly in chats or widgets, without having to visit the original website . This is detrimental to creators, who are losing a portion of their audience and, with it, their revenue.
That’s why Cloudflare is now seeking to leverage its position as an infrastructure player to prevent the market from completely shifting to a few dominant platforms. The company has traditionally focused on accelerating and securing web traffic, but amid the exponential growth of generative AI, Prince is increasingly focusing on preserving the internet as a space where projects of all sizes, from local media to small communities and creators, can survive and thrive.
He also criticized Google’s policies regarding its search and AI crawlers. In an effort to achieve leadership in AI, Google combined the crawler that indexes websites for search with the crawler that collects data for AI. As a result, if a site blocks Google’s AI bot, it is automatically removed from search indexing . This creates a trap for publishers and developers: they don’t want their content to be uncontrolledly included in training sets, but they can’t afford to lose their Google search rankings, which account for a significant portion of their traffic.
According to Prince, you can’t leverage your monopoly on yesterday’s market to gain a foothold in tomorrow’s. He directly identifies Google as the main obstacle to the Internet’s development and believes the company should separate its search and artificial intelligence crawlers so that websites can manage access independently.
Cloudflare also revealed previously unpublished statistics on Google’s privileged access to the web compared to other AI players . The company estimates that Google currently sees 3.2 times more pages than OpenAI, 4.6 times more than Microsoft, and 4.8 times more than Anthropic or Meta. Essentially, Google has a unique view of the internet, which strengthens its position in both search and AI.
At the same time, Prince says, the initial results of blocking AI crawlers for publishers and other content owners look promising. Unique human content—from local newsrooms sensitive to regional trends to Reddit users sharing their strangest or most profound thoughts— remains an extremely valuable raw material for AI models . This means creators have the opportunity to build new models for a fee, from licensing to specialized partnerships with AI companies.
In the future, Prince acknowledges, regulating AI and content access may require legislative intervention. For now, Cloudflare is trying to exert pressure on the market through its decisions and agreements to ensure that new AI business models are more pluralistic, expand the market, and do not concentrate power in the hands of one or two players. There’s nothing romantic about this, he says: it simply means greater growth opportunities for everyone, which means Cloudflare itself will have more customers and a broader internet to protect.
Prince compared the situation to a superhero franchise: the hero of the last film becomes the villain of the next. In his interpretation, Google has become the “bad guy” holding the internet back . And until the company agrees to play by the same rules as everyone else— primarily by separating its search engine from its AI crawler —it will be extremely difficult to completely block unwanted content.
Redazione