On July 1, 2026, Cloudflare announced a default that will reshape traffic for content sites: starting September 15, 2026, pages that display ads will block "mixed-use" AI crawlers by default. At the same time, the earlier Pay Per Crawl experiment is being widened into a Pay Per Use model.

Who Gets Blocked, and Who Doesn't

Cloudflare splits crawler behaviour into three categories, and that split is the key to understanding the change:

  • Search: collecting and indexing content to answer questions later — still allowed.
  • Agent: automated behaviour acting in real time on a person's behalf — blocked by default on ad-bearing pages.
  • Training: taking content to train or fine-tune a model — blocked by default on ad-bearing pages.

The new defaults apply to new customers, to new sites added by existing customers, and to all existing free-tier customers. In other words, plenty of independent site owners will see behaviour change on September 15 without touching anything. Site owners can override the defaults in their settings. Launch partners for Pay Per Use include Ceramic.ai (paying publishers when content shows up in search results) and You.com (paying when an agent accesses premium content).

What It Means for a Self-Hosted Site

If you run ads and a meaningful share of your exposure comes from AI answers and AI agents, this default can cut your server load and your visibility at the same time. The bandwidth bill looks better; the odds of being cited in an AI answer go down — and citation is quietly becoming a traffic channel of its own. If you don't run ads and monetise through your own product or service, the default won't change your situation automatically, and deciding who to allow is on you.

Four Things Worth Doing Now

  • Measure before you decide: from Nginx/Apache logs, break down requests and bandwidth by User-Agent for GPTBot, ClaudeBot, PerplexityBot, CCBot and Bytespider, and compare that cost against the referrals each actually sends back.
  • Tier access instead of slamming the door: allow retrieval crawlers, which are still a visibility channel; restrict pure training crawlers; disallow admin paths, member areas and duplicated filter pages outright.
  • Keep a control point of your own: robots.txt is a gentleman's agreement. Use Nginx limit_req and limit_conn keyed on User-Agent or IP, and return 403 where warranted, rather than delegating the whole policy to an upstream default.
  • Watch referral data around the date: capture a baseline before and after September 15, then decide whether to keep blocking or open back up.

Why This Makes a VPS More Valuable

Once "who is allowed to crawl me" becomes an active business decision, you need access to the layer where that is configured. On shared hosting you can't write rate-limit rules, can't see full logs, and can't apply per-UA policy. On your own VPS, all of it is yours. SharkCloud plans include dedicated IPs and clearly stated resource specs, with Hong Kong, Japan and US nodes for hosting close to your audience — so the switch stays in your hands.