OpenAI Will Honor Shared robots.txt Signal for AI Training Preferences
OpenAI says it will honor a common signal in robots.txt that lets websites state their preferences on the use of their content for foundational generative AI training. The company describes it as a voluntary effort that allows operators to communicate one preference to multiple participating providers at once.
- OpenAI will honor a common robots.txt signal that expresses website preferences on foundational generative AI training.
- The voluntary effort lets operators communicate one preference to multiple participating providers instead of maintaining separate rules per crawler.
- OpenAI aims to implement the signal in the coming months and will share details later.
- Its existing robots.txt crawler controls will continue to apply and be respected.
- The signal draws on the IETF AI Preferences working group, and OpenAI encourages other AI providers to honor it.
One signal instead of many crawler rules
According to OpenAI, website operators have until now had to identify individual crawlers and maintain separate rules for each and every provider. The shared signal is meant to cut that work. With a common signal in robots.txt, a publisher can express the same preference to several participating AI companies in one place.
OpenAI describes the effort as voluntary. The signal applies to the use of content for foundational generative AI training, and it works through robots.txt, the file websites already use to communicate with automated crawlers.
Timing and existing controls
The company has not yet put the signal into practice. OpenAI says it aims to implement it "in the coming months" and will share another update with details about its implementation when it does so. No specific date was given.
The new signal does not replace what is already available. OpenAI states that its existing crawler controls, which give web publishers flexibility and control over their pages, will continue to apply and be respected. Operators who have set rules for individual OpenAI crawlers therefore do not lose those options.
Link to the IETF standards work
The move draws on the ongoing work of the Internet Engineering Task Force (IETF) AI Preferences working group. OpenAI says it is joining other participants in the working group in announcing the shared signal, and believes this strengthens its continued participation in the IETF standards discussions.
The company adds that it will continue to adapt its practices as those conversations and the wider landscape evolve, in view of appropriate controls and fairness.
What it means for publishers and other AI providers
For publishers and website operators, the practical benefit described by OpenAI is less maintenance: one signal that several providers can read, rather than a growing list of provider-specific rules. The benefit depends on how many companies take part, since the effort is voluntary and covers only participating providers.
OpenAI calls the step "a practical step toward an open web" where people can publish and discover information, and where website owners have clearer, simpler ways to communicate their choices about how their content is used. The company also encourages other AI providers to honor the signal.
