CRYPTOREPORTCLUB
  • Crypto news
  • AI
  • Technologies
Wednesday, July 2, 2025
No Result
View All Result
CRYPTOREPORTCLUB
  • Crypto news
  • AI
  • Technologies
No Result
View All Result
CRYPTOREPORTCLUB

Cloudflare Hits the Kill Switch on AI Crawlers—And an Entire Industry Cheers

July 1, 2025
152
0

Cloudflare CEO Matthew Prince just made it official: Starting today, Cloudflare will block by default known AI bots from scraping sites unless given explicit permission—or unless they pay.

It’s not just a technical tweak. Cloudflare’s “Content Independence Day” marks a collective escalation. More than a million websites had already opted into blocking since last fall, but now it’s the default for every new Cloudflare domain. Furthermore, a roster of media heavyweights—from the Associated Press, Time, The Atlantic, and BuzzFeed to Reddit, Quora, and Universal Music Group—have signed on.

Related Post

Bitcoin Bulls Should Be Wary as Dollar Index Chart Flashes ‘Death Cross’: Technical Analysis

July 2, 2025
Pudgy Penguins surges 64% in a week — What’s driving PENGU token rally?

Pudgy Penguins surges 64% in a week — What’s driving PENGU token rally?

July 2, 2025

The movement has been well-received on social media, with investors and SEO experts praising the initiative.

“This is the way,” said said Ed Newton-Rex, former VP of audio at Stability AI; “Cloudflare solving problems again,” said pseudonymous trader and content creator Romano RNR. “It'll be OK. A force is needed to tilt the balance back to where it once was,” said former Google and current SEO consultant Pedro Dias.



Of course, not everyone is convinced. The main argument against such a move is that it could hurt businesses not appearing on AI searches.

Or maybe not: Prince pointed out that a decade ago, Google crawled two pages for every one visit it sent to a publisher. Given that the number of people using the internet has skyrocketed to include most of the planet, publishers should benefit. But they haven’t, since in order to be “seen” on Google, publishers need to make their content easier for the search giant to summarize—the better to keep them within Google Search.

So today, for every “18 pages that Google takes from you, you get one visitor,” Prince said.

But the numbers for AI companies are exponentially more devastating. Said Prince: “What's the ratio for OpenAI six months ago? 250 to 1. What is it today? 1,500 to 1.” And OpenAI is hardly the worst offender. Anthropic was “6,000 to 1 six months ago. What is it today? 60,0000” to 1, he said.

“If the internet is going to survive the age of AI, we need to give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself,” Prince argued. “I'm not going to sugarcoat it. If people don't have incentive to create content, they're not going to create content” he said separately. “We have to give that back or the internet dies,” Prince said.

“Pay-per-crawl”—or don’t crawl

To pressure AI companies into putting up or getting shut out, Cloudflare is pioneering a new approach to AI content access: a permissioned, pay-to-crawl marketplace. For the first time, publishers can set their own rates for AI bots—invoking the long-dormant HTTP 402 “payment required” code as a digital tollbooth.

Publishers can allow free access, charge for each crawl, or block them entirely. The company says this is only available for leading content creators in early access, but the plan is to scale the system.

The “pay-per-crawl” initiative arrives as some of the web’s biggest content providers seek leverage over AI companies, after years of having content ingested for free and monetized elsewhere. “When AI companies can no longer take anything they want for free, it opens the door to sustainable innovation built on permission and partnership,” said Condé Nast CEO Roger Lynch.

This coalition-building, Prince believes, is essential.

“You cannot have a good sort of market without scarcity, which is why this needs to be a collective effort,” he said. He cautioned that licensing deals made today, if not coupled with real blocking, would be worse in the future.

Research confirms that the movement is more than symbolic. According to studies by Originality.AI and the Reuters Institute, 48% of the top global websites are already blocking AI crawlers. Last year, Cloudflare implemented an option to make it easy to block AI crawlers, and more than a million customers have exercised that option, the company said.

Now, new domains signing up with Cloudflare are asked upfront whether they want to allow AI crawlers, with blocking as the default. Existing customers can change their settings anytime.

Edited by Andrew Hayward

Share212Tweet133ShareShare27ShareSend

Related Posts

Crypto news

Bitcoin Bulls Should Be Wary as Dollar Index Chart Flashes ‘Death Cross’: Technical Analysis

July 2, 2025
0

This is a daily technical analysis by CoinDesk analyst and Chartered Market Technician Omkar Godbole. Bitcoin BTC$107,789.68 bulls are optimistic that the U.S. dollar's broader sell-off will continue in the second half of the year, fueling the crypto bull run. However, one chart warrants caution when relying on bearish dollar...

Read moreDetails
Pudgy Penguins surges 64% in a week — What’s driving PENGU token rally?

Pudgy Penguins surges 64% in a week — What’s driving PENGU token rally?

July 2, 2025
Unexpected Breakout from Two Altcoins Waiting to Be Delisted! They Rise Over 100 Percent!

Unexpected Breakout from Two Altcoins Waiting to Be Delisted! They Rise Over 100 Percent!

July 2, 2025

XRP Enters Grayscale’s NYSE-Listed Digital Asset ETF, Eyes on Pending 17 XRP ETF Filings

July 2, 2025

Figma Eyes Further Bitcoin Exposure in Run-Up to Hopeful NYSE Debut

July 2, 2025
Top 5 Tokenized Real Estate Platforms Transforming Property Investment

Top 5 Tokenized Real Estate Platforms Transforming Property Investment

July 1, 2025

Moon or Doom: Will Bitcoin See Fireworks on July 4?

July 1, 2025

Recent News

SoftBank’s acquisition of AI chip designer Ampere may be facing an FTC probe

SoftBank’s acquisition of AI chip designer Ampere may be facing an FTC probe

July 2, 2025

Bitcoin Bulls Should Be Wary as Dollar Index Chart Flashes ‘Death Cross’: Technical Analysis

July 2, 2025
Prime Day deals include the Apple Watch Series 10 for $100 off

Prime Day deals include the Apple Watch Series 10 for $100 off

July 2, 2025
The best laptop you can buy in 2025

The best laptop you can buy in 2025

July 2, 2025

TOP News

  • Apple details new fee structures for App Store payments in the EU

    Apple details new fee structures for App Store payments in the EU

    540 shares
    Share 216 Tweet 135
  • Top 5 Tokenized Real Estate Platforms Transforming Property Investment

    535 shares
    Share 214 Tweet 134
  • Buying Art from a Gallery. A Guide to Making the Right Choice

    534 shares
    Share 214 Tweet 134
  • New Pokémon Legends: Z-A trailer reveals a completely large model of Lumiose Metropolis

    564 shares
    Share 226 Tweet 141
  • Bitcoin Bullishness For Q3 Grows: What Happens In Every Post-Halving Year?

    534 shares
    Share 214 Tweet 134
  • About Us
  • Contact Us
  • Privacy Policy
  • Terms of Use
Advertising: digestmediaholding@gmail.com

Disclaimer: Information found on cryptoreportclub.com is those of writers quoted. It does not represent the opinions of cryptoreportclub.com on whether to sell, buy or hold any investments. You are advised to conduct your own research before making any investment decisions. Use provided information at your own risk.
cryptoreportclub.com covers fintech, blockchain and Bitcoin bringing you the latest crypto news and analyses on the future of money.

© 2023-2025 Cryptoreportclub. All Rights Reserved

Welcome Back!

Login to your account below

Forgotten Password?

Retrieve your password

Please enter your username or email address to reset your password.

Log In
No Result
View All Result
  • Crypto news
  • AI
  • Technologies

Disclaimer: Information found on cryptoreportclub.com is those of writers quoted. It does not represent the opinions of cryptoreportclub.com on whether to sell, buy or hold any investments. You are advised to conduct your own research before making any investment decisions. Use provided information at your own risk.
cryptoreportclub.com covers fintech, blockchain and Bitcoin bringing you the latest crypto news and analyses on the future of money.

© 2023-2025 Cryptoreportclub. All Rights Reserved