CRYPTOREPORTCLUB
  • Crypto news
  • AI
  • Technologies
Friday, July 25, 2025
No Result
View All Result
CRYPTOREPORTCLUB
  • Crypto news
  • AI
  • Technologies
No Result
View All Result
CRYPTOREPORTCLUB

Cloudflare Hits the Kill Switch on AI Crawlers—And an Entire Industry Cheers

July 1, 2025
152
0

Cloudflare CEO Matthew Prince just made it official: Starting today, Cloudflare will block by default known AI bots from scraping sites unless given explicit permission—or unless they pay.

It’s not just a technical tweak. Cloudflare’s “Content Independence Day” marks a collective escalation. More than a million websites had already opted into blocking since last fall, but now it’s the default for every new Cloudflare domain. Furthermore, a roster of media heavyweights—from the Associated Press, Time, The Atlantic, and BuzzFeed to Reddit, Quora, and Universal Music Group—have signed on.

Related Post

Public Shell Firms Ramping Up Altcoin Buys Draws Skepticism: FT

July 25, 2025

Bitcoin to Hit $135K by Year-End in Base-Case Forecast, $199K in Bullish Scenario: Citi

July 25, 2025

The movement has been well-received on social media, with investors and SEO experts praising the initiative.

“This is the way,” said said Ed Newton-Rex, former VP of audio at Stability AI; “Cloudflare solving problems again,” said pseudonymous trader and content creator Romano RNR. “It'll be OK. A force is needed to tilt the balance back to where it once was,” said former Google and current SEO consultant Pedro Dias.



Of course, not everyone is convinced. The main argument against such a move is that it could hurt businesses not appearing on AI searches.

Or maybe not: Prince pointed out that a decade ago, Google crawled two pages for every one visit it sent to a publisher. Given that the number of people using the internet has skyrocketed to include most of the planet, publishers should benefit. But they haven’t, since in order to be “seen” on Google, publishers need to make their content easier for the search giant to summarize—the better to keep them within Google Search.

So today, for every “18 pages that Google takes from you, you get one visitor,” Prince said.

But the numbers for AI companies are exponentially more devastating. Said Prince: “What's the ratio for OpenAI six months ago? 250 to 1. What is it today? 1,500 to 1.” And OpenAI is hardly the worst offender. Anthropic was “6,000 to 1 six months ago. What is it today? 60,0000” to 1, he said.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Prince argued. “I'm not going to sugarcoat it. If people don't have incentive to create content, they're not going to create content” he said separately. “We have to give that back or the internet dies,” Prince said.

“Pay-per-crawl”—or don’t crawl

To pressure AI companies into putting up or getting shut out, Cloudflare is pioneering a new approach to AI content access: a permissioned, pay-to-crawl marketplace. For the first time, publishers can set their own rates for AI bots—invoking the long-dormant HTTP 402 “payment required” code as a digital tollbooth.

Publishers can allow free access, charge for each crawl, or block them entirely. The company says this is only available for leading content creators in early access, but the plan is to scale the system.

The “pay-per-crawl” initiative arrives as some of the web’s biggest content providers seek leverage over AI companies, after years of having content ingested for free and monetized elsewhere. “When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Condé Nast CEO Roger Lynch.

This coalition-building, Prince believes, is essential.

“You cannot have a good sort of market without scarcity, which is why this needs to be a collective effort,” he said. He cautioned that licensing deals made today, if not coupled with real blocking, would be worse in the future.

Research confirms that the movement is more than symbolic. According to studies by Originality.AI and the Reuters Institute, 48% of the top global websites are already blocking AI crawlers. Last year, Cloudflare implemented an option to make it easy to block AI crawlers, and more than a million customers have exercised that option, the company said.

Now, new domains signing up with Cloudflare are asked upfront whether they want to allow AI crawlers, with blocking as the default. Existing customers can change their settings anytime.

Edited by Andrew Hayward

Share212Tweet133ShareShare27ShareSend

Related Posts

Crypto news

Public Shell Firms Ramping Up Altcoin Buys Draws Skepticism: FT

July 25, 2025
0

A number of public companies may be about to build hoards of altcoins in an attempt to pump their share prices. Such companies are attempting to replicate the model mastered by Michael Saylor's Strategy (MSTR) which has now accumulated 2.9% of all the bitcoin BTC$116,623.03 that will ever exist. Since...

Read moreDetails

Bitcoin to Hit $135K by Year-End in Base-Case Forecast, $199K in Bullish Scenario: Citi

July 25, 2025

The Long-Awaited Altcoin’s Release Date Has Finally Been Announced: Airdrop Hunters Were Also Following Closely

July 25, 2025

Crypto Market Retreats Amid Record Liquidity and Signs of Froth

July 25, 2025

Pundit Sees XRP Hitting $20 to $30 Top, But Here’s Why He Thinks a Correction Could Follow

July 25, 2025
US crypto legislation drives $4B surge in stablecoin supply

US crypto legislation drives $4B surge in stablecoin supply

July 25, 2025

Cognition’s AI Developer ‘Devin’ Eyes $10 Billion Valuation—Even If It’s Not Mindblowing Yet

July 24, 2025

Recent News

New Jersey is the latest state to embrace digital IDs

New Jersey is the latest state to embrace digital IDs

July 25, 2025
AI will soon be able to audit all published research—what will that mean for public trust in science?

AI will soon be able to audit all published research—what will that mean for public trust in science?

July 25, 2025
Internet Archive is now an official US government document library

Internet Archive is now an official US government document library

July 25, 2025

Public Shell Firms Ramping Up Altcoin Buys Draws Skepticism: FT

July 25, 2025

TOP News

  • Bitcoin Sees Long-Term Holders Sell As Short-Term Buyers Step In – Sign Of Rally Exhaustion?

    Bitcoin Sees Long-Term Holders Sell As Short-Term Buyers Step In – Sign Of Rally Exhaustion?

    534 shares
    Share 214 Tweet 134
  • The AirPods 4 are still on sale at a near record low price

    533 shares
    Share 213 Tweet 133
  • Ripple Partners With Ctrl Alt to Expand Custody Footprint Into Middle East

    533 shares
    Share 213 Tweet 133
  • Cyberpunk 2077: Ultimate Edition comes to the Mac on July 17

    533 shares
    Share 213 Tweet 133
  • HBO confirms The Last of Us season 3 will arrive in 2027

    533 shares
    Share 213 Tweet 133
  • About Us
  • Contact Us
  • Privacy Policy
  • Terms of Use
Advertising: digestmediaholding@gmail.com

Disclaimer: Information found on cryptoreportclub.com is those of writers quoted. It does not represent the opinions of cryptoreportclub.com on whether to sell, buy or hold any investments. You are advised to conduct your own research before making any investment decisions. Use provided information at your own risk.
cryptoreportclub.com covers fintech, blockchain and Bitcoin bringing you the latest crypto news and analyses on the future of money.

© 2023-2025 Cryptoreportclub. All Rights Reserved

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Crypto news
  • AI
  • Technologies

Disclaimer: Information found on cryptoreportclub.com is those of writers quoted. It does not represent the opinions of cryptoreportclub.com on whether to sell, buy or hold any investments. You are advised to conduct your own research before making any investment decisions. Use provided information at your own risk.
cryptoreportclub.com covers fintech, blockchain and Bitcoin bringing you the latest crypto news and analyses on the future of money.

© 2023-2025 Cryptoreportclub. All Rights Reserved