Mingpai: OpenAI publishes GPTBot, a web crawler tool with "identity"

CTOnews.com August 8 news, OpenAI yesterday released its web crawler tool GPTBot. Officials claim that the GPTBot tool can train OpenAI's AI models by collecting web information in a transparent way based on copyright.

OpenAI said that GPTBot uses a proprietary web page UA to indicate its crawler identity, and the full UA string is (Mozilla / 5.0AppleWebKit / 537.36 / KHTML, like Gecko; compatible; GPTBot / 1.0; + https://openai.com/ gptbot), and any webmaster is free to allow or prevent the crawler tool from collecting data.

▲ diagram source OpenAI

▲ image source OpenAIOpenAI claims that if the webmaster does not want to be collected by the crawler, the administrator can completely prohibit GPTBot from grabbing information in the robots.txt file of the website server, or GPTBot can grab the specified information on the site at his own discretion.

OpenAI has previously been criticized by the industry for the "invasion of privacy". Now OpenAI launches such a set of GPTBot crawler tools in response to external criticism, and can also promote the industry to establish benchmarks related to AI training crawler tools. CTOnews.com previously reported that OpenAI recently announced the registration of the GPT-5 trademark, the GPTBot crawler tool is also expected to help GPT-5 related model training.