Cloudflare and OpenAI pilot adds hundreds of millions of pages to ChatGPT search

What the pilot found

Cloudflare says the pilot used a participating cohort of ~1% of its total network. In that scope, it enabled OpenAI to discover and index hundreds of millions of new web pages. Cloudflare describes the early answer to its central question as "a resounding yes", and says the added pages give ChatGPT more sources to draw on when people look for relevant, timely information.

The company argues that such pages can be valuable even when few other sites link to them. Its examples include the manual for a specific washing machine model, current passport requirements and fees, or a local restaurant's latest menu and opening hours. Finding the right page, Cloudflare says, can be the difference between a wrong answer and a helpful one.

According to Cloudflare, its network provides signals that help OpenAI's search crawler find more useful pages. OpenAI then evaluates and ranks how those pages appear on its own, based on relevance and helpfulness. ChatGPT Search links back to relevant sources, which gives people a way to reach the original content and the publishers behind it.

Filtering out wasted crawling and flagging harmful pages

Cloudflare says finding a URL is only the first step. The team also needs to know whether a page is accessible, whether it offers anything distinct, and what it might help someone do. One lesson from the pilot is how easily automated traffic can make a nonexistent page look interesting. Bots repeatedly request files that are not there, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content.

Recognizing these patterns lets crawlers cut wasted work before fetching a page, which Cloudflare says benefits site owners. The company also describes change signals that tell a crawler when a page is worth another visit and when it is not. A crawler that repeatedly downloads an unchanged page adds load without any benefit, the post notes.

Cloudflare is also testing signals that identify potentially harmful pages. It gives the example of a legitimate site with a single compromised page, such as a fake login form planted by an attacker. Flagging that one page rather than the whole domain protects people without penalizing the site owner. Early results suggest these signals can help identify pages that should be kept out of an index.

There is a trade-off. Finding new pages can initially increase crawling while those pages are evaluated and indexed. The stated goal is to keep discovering and stay current while reducing the work this takes.

Next steps and what it means for site owners

The next phase of the pilot will improve the selection of pages, test faster and more reliable change signals, and measure what happens after content is discovered. Cloudflare says it is working with OpenAI to understand how better discovery can translate into real business outcomes, and that it will keep sharing results.

For Cloudflare customers, the company says the aim is less unnecessary crawler load. For OpenAI, it means more of its search resources go to information that helps people. Cloudflare also links the work to OpenAI's efforts with GPT-6 and Intelligent UI, saying the pilot complements them by helping ChatGPT find more up-to-date information from across the web.

On control, Cloudflare stresses that domain owners' choices matter throughout. A signal about a page does not grant permission to access it, and the research uses metadata about the web for search and discovery rather than sharing page content for model training. Site owners can use Cloudflare's AI Crawl Control to understand and manage how AI systems access their websites.