Cloudflare published two crawler-related posts back to back this month. The October 16 post announced that it now continuously evaluates whether Verified bots behave: verifying identity only tells you who a bot is, while trust depends on what it does next. The October 15 post shared early results from a research pilot with OpenAI that began in July 2026. Both bear directly on how site owners deal with crawlers.
The Trust Score Looks at Two Things
- Does the bot respect robots.txt? The system checks whether a bot's observed traffic over time is consistent with the robots.txt preferences a site publishes (per RFC 9309). Isolated, one-off violations matter little; what counts are repeated patterns of disregarding site preferences. Cloudflare also points out that robots.txt sets only the minimum rules — anything it doesn't forbid is not thereby allowed.
- Does it slow down after an HTTP 429? A 429 (Too Many Requests) is a site's direct signal to reduce the request rate. The system checks whether the bot slows down, pauses, or carries on at the same pace — consistently slowing down raises trust, while repeatedly ignoring the signal lowers it.
- The result is an explainable "trust band" rather than a permanent judgment about a bot. Cloudflare is already using the system internally, will share the scores publicly on Radar, and plans to offer it to all customers, who will be able to choose the trust threshold used to decide how Verified bot traffic is handled. As an example, the post recalls that in 2025 it de-listed Perplexity's bot from the Verified bot list after it tried to get around sites' controls.
The OpenAI Pilot: Helping ChatGPT Find Pages Nobody Links To
- The pilot started in July 2026 with a participating cohort of about 1% of Cloudflare's network. Cloudflare provides signals from its network to OpenAI's search crawler, and OpenAI evaluates and ranks the pages independently. So far this has let OpenAI discover and index hundreds of millions of new web pages.
- Cloudflare says the research uses metadata about pages for search and discovery, not page content for model training, and that a signal about a page doesn't grant permission to access it; site owners' choices still apply.
- The post also notes that finding new pages can initially increase crawling while they are evaluated and indexed. It found that many sites redirect almost any address to their homepage, or serve the same content at several addresses, which wastes crawler work; filtering these out reduces the load on origin servers.
What Site Owners Can Do Now
Keep robots.txt accurate — it is now both the rulebook for search engines and evidence Cloudflare uses to judge whether a crawler can be trusted — and return a real 404 for pages that don't exist instead of redirecting everything to the homepage. If your server genuinely can't keep up, return 429 or 503 with Retry-After; how to do it and what it costs are covered in Google's October 6 rewrite of its emergency crawl-rate guide. To find out who is crawling first, read the access logs as described in diagnosing 100% CPU usage on a VPS; for the bandwidth impact of AI crawlers see this piece, for Cloudflare's default blocking of AI crawlers see the September 15 change, and to put your site behind Cloudflare see speeding up and protecting a VPS website with Cloudflare.