On October 6, 2026, Google rewrote its "Reduce the Google crawl rate" documentation. According to the changelog, it restructured the emergency crawl-rate reduction section and put explanations and examples of the Retry-After header straight into the guide. The changelog also notes that support for Retry-After is not new — it was already documented on the "Temporarily pause or disable a website" page; this just puts it where people will find it. If you run a site on a small machine, it is the official brake worth bookmarking.

Find the Cause Before You Brake

Based on reports Google has received, the documentation says the most common causes of a sudden jump in crawling are the site's own URL structures: faceted navigation and other sorting and filtering features, calendars that generate a lot of URLs for specific dates, and Dynamic Search Ad targets. Google recommends checking with your hosting company and your server's recent access logs to see whether the traffic matches one of these. For a way into the logs, see diagnosing 100% CPU usage on a VPS; if the traffic actually comes from AI crawlers or bots pretending to be crawlers, that is a different problem, covered in AI crawlers are eating your server bandwidth.

The Emergency Brake: Return 500, 503 or 429

  • How: for a short period (the documentation's examples are a couple of hours, or 1-2 days), return 500, 503 or 429 to Google's crawl requests instead of 200.
  • Effect: when Google's crawlers meet a significant number of these status codes, they reduce the crawl rate across the whole hostname (for example subdomain.example.com), including both the URLs that return errors and those that return content, and automatically raise it again once the errors fall.
  • Retry-After: with 503 or 429 you can add a Retry-After header, either as a delay in seconds (such as Retry-After: 120) or as an absolute UTC date and time, to tell crawlers when to come back.

The Costs Are Spelled Out Too

  • For Search, Googlebot discovers fewer new pages while the brake is on, existing pages are refreshed less often (changes to prices and availability, for example, take longer to show up), and removed pages may stay in the index longer. For Google Ads, campaigns may be cancelled or paused and ads may not serve.
  • Don't keep it up for more than 1-2 days. If Googlebot sees these status codes on the same URL for several days, that URL may be dropped from the index.
  • If serving errors to crawlers isn't feasible, you can file a special request about an unusually high crawl rate, but you cannot ask for a higher rate, and evaluation may take several days.

Putting It Into Practice on a VPS

Serve the 503 at the web server or application layer, aimed only at crawlers and only for the emergency, and remember to remove it afterwards. User-Agent strings can be spoofed; Google documents how to verify its crawlers with a reverse DNS lookup or against its published IP addresses. If the load is already causing 502 / 504 errors, bring the service back first using the 502 / 504 troubleshooting guide. In the longer run, if the machine can't cope with normal crawling, it is time to re-size the VPS for the workload or put static assets behind a CDN.