Google published two crawling-related guides in one week. On November 11 a Search Central blog post explained how to move a site to a CDN without overwhelming the origin server or locking out Googlebot, and on November 17 the crawling documentation gained an "Allow Google-Agent traffic" page that explains how to let it through CDNs, firewalls and bot-detection services. If you host a site on a VPS behind a CDN or a WAF, both are worth a read.

Why the origin gets busier after moving to a CDN

  • Google sets its crawl capacity limit (hostload) by how fast and how stable your server is: when responses get slower it slows down, and when they are fast and consistent, and there is crawl demand, the limit gradually goes up.
  • Once your domain points to a CDN, connection and response times drop sharply, which to Google looks like your server just got a hardware upgrade, so the crawl rate starts to climb.
  • But a new CDN starts with an empty cache. Human visitors cluster on a few pages such as the homepage and popular articles, which get cached quickly; crawlers do the opposite and work through large numbers of URLs one by one, and every URL not yet cached passes through the CDN to the origin. Until the caches warm up, the CDN is not yet a shield and the origin still does the heavy lifting.

Google's three recommendations

  • If you block bots, allow verified crawlers in your CDN and firewall. Most CDNs and WAFs have a toggle or managed rule to allow known or verified bots, and Google's crawlers can be identified by the IP ranges it publishes; better still, target specific unwanted bots with robots.txt or rules instead of blocking all automated traffic. If you turn on site-wide CAPTCHA challenges without exemptions, Google's crawlers cannot pass them, and over time your pages drop out of search results.
  • Pre-warm your caches before switching: before routing all traffic through the new CDN, request your highest-traffic pages, pages about to hit a traffic peak (such as promotions) and your most frequently crawled URLs through the CDN at a controlled pace.
  • If the origin genuinely cannot cope, return 429 or 503 for a short window (a few hours, or 1 to 2 days at most) to slow crawlers down; details are in our report on Google's guide to urgently reducing the crawl rate.

What Google-Agent is and how to allow it

Google-Agent is the name used by AI agents hosted on Google's infrastructure when they browse the web and perform actions at a user's request. It is a user-triggered fetcher, and Google's documentation says that because such fetches are requested by a user, these fetchers generally ignore robots.txt rules. Whether to allow it is your call; Google's page is written for site owners who want to let it in. You identify it by a User-Agent header containing Google-Agent and a source IP within the ranges in Google's published user-triggered-agents.json; Google is also experimenting with Web Bot Auth, signing requests with HTTP Message Signatures (RFC 9421) and adding a Signature-Agent: g="https://agent.bot.goog" header. If requests are still blocked after you allow it, Google lists these things to check:

  • Rule order: the allow rule must come before broad cloud or data-center ASN blocks, managed bot rules and category rules such as "block AI agents".
  • Challenge types: skip interactive CAPTCHAs and JavaScript challenges, not just outright blocks, including verification widgets that appear on search boxes or form submission.
  • Coverage: allow more than the homepage, including on-site search, filters, detail pages, add-to-cart, booking and the XHR, GraphQL and api.* endpoints behind them.
  • Per-IP rate limits: requests from many users share the same egress IPs, so per-IP rate-limiting rules (including tools you run yourself, such as fail2ban) should exempt these ranges.
  • IP updates and proxies: keep IP rules in sync with user-triggered-agents.json, and make sure load balancers and reverse proxies preserve the original client IP, the User-Agent and the signature-related headers.

For CDN basics, see what a CDN is, and for setting up Cloudflare, see speeding up and protecting your site with Cloudflare.