OpenAI to Support Common Signal for AI Training Preferences

OpenAI announced that it will honor a common signal in robots.txt, the file used by websites to express preferences about how their content is used for foundational generative AI training. This voluntary effort aims to make it easier for publishers and site operators to convey a single preference to multiple participating AI providers at once.

Under previous arrangements, website operators had to manually identify individual web crawlers and maintain separate sets of rules for each provider. Managing separate instructions created extra work for operators who wanted to manage how AI companies accessed their sites. Providing a shared signal across multiple participating providers reduces that effort by allowing site owners to state their choices across providers.

OpenAI aims to implement the signal in the coming months. The company stated that it will share another update with implementation details when the deployment occurs. This new signal will not replace the existing controls OpenAI currently provides through robots.txt. The company's existing crawler controls, which offer web publishers flexibility and control over their pages, will continue to apply and be respected.

The initiative draws on ongoing work by the Internet Engineering Task Force AI Preferences working group. OpenAI is joining other participants in the working group for the announcement of the shared signal. The company believes joining this announcement strengthens its continued participation in Internet Engineering Task Force standards discussions. OpenAI stated it will continue to adapt its practices as those standards conversations evolve.

The company expressed its view that adopting the signal represents a practical step toward an open web where people publish and discover information while website owners have clearer ways to communicate choices. OpenAI is also encouraging other AI providers to honor the signal.