The war over who controls internet content just hit a new milestone. Cloudflare CEO Matthew Prince announced that his company blocked more than 416 billion AI bot requests since July, when the tech giant made blocking AI crawlers the default option for website owners.
That huge number follows Cloudflare’s recent launch of Content Independence Day, which gave site owners control over whether AI companies could scrape their content. The kicker? Now AI companies have to pay if they want in.
Prince framed the stakes in a recent interview with Wired, underlining how artificial intelligence is fundamentally revising the internet’s economic premise. “The business model of the internet has always been to generate content that drives traffic and then sell either things, subscriptions, or ads,” he said. “What I think people don’t realize, though, is that AI is a platform shift. The business model of the internet is about to change dramatically.”
Google Combines Search and AI Crawlers, Giving Sites No Option to Opt Out
While Cloudflare successfully blocks nearly all AI crawlers, there is one glaring exception that is giving website owners headaches: Google. The search giant made the controversial decision to combine its traditional search crawler with its AI crawler into a single bot. That means websites that block Google’s AI bot disappear from regular Google search results entirely.
“You can’t opt out of one without opting out of both, which is a real challenge – it’s crazy,” Prince said. “It shouldn’t be that you can use your monopoly position of yesterday in order to leverage and have a monopoly position in the market of tomorrow.”
This puts website owners in an impossible bind. They can either allow Google’s AI to freely scrape their content or risk losing visibility in the world’s most dominant search engine. For most businesses, that’s not really a choice at all.
The push to protect human-created content is not solely about the money. Studies have continually indicated that AI models trained on data created essentially by themselves quickly start to degrade and produce what experts commonly refer to as “slop.” Human-generated content remains crucial in training capable AI systems.
Meanwhile, AI-powered summaries are already cutting into website traffic, hitting publications dependent on page views and ad revenue particularly hard. One potential solution is for AI companies to license content from creators and allow publishers to realize revenue from their work even as AI changes how people consume information.
It’s not purely altruism driving Prince to fight for these content creators. Cloudflare has a business interest in keeping the internet ecosystem diverse and thriving. More websites mean more potential customers who need protection and content delivery services.
The Market Share of CloudFlare and the Fragility of Modern Internet Infrastructure
The strategy seems to be paying off. By 2022, Cloudflare controlled about 80% of the content delivery network market, easily one of the most dominant positions in internet infrastructure.
Yet Cloudflare’s enormous market share is a vulnerability, too. A single misconfigured file at Cloudflare knocked out a swath of the internet in November, in a harbinger of how dependent the world has become on just a few companies.
The modern web basically runs on a handful of tech giants: Amazon Web Services, Microsoft Azure, Cloudflare, CrowdStrike, and Google. The good thing with this concentration is that it makes the internet more efficient and streamlined for business. But it has also created huge single points of failure.
And when each of those services goes down, the ripple effects are felt around the world. Companies lose billions in revenue, critical services go down, and millions of people find themselves suddenly disconnected from the digital services they rely on daily.
As Prince noted, nobody knows what the business model of the internet will ultimately look like. What’s clear is that the current tension between AI companies hungry for training data and content creators seeking to protect their work represents just the beginning of a much larger transformation.
The choices today regarding who has control over Web content and under what conditions it is shared with AI companies will help frame the Internet for decades to come. With 416 billion blocked requests and counting, Cloudflare has firmly planted its flag on the side of giving creators control over their own content. Whether that’s enough to build a sustainable model for the AI era remains to be seen.
