Get the App
SLTechnology News&Howtos  ›  IT Information  › 

OpenAI now allows websites to prevent their web crawlers from grabbing data, preventing data from being used to train AI models

Shulou Source: shulou.com Published: 2023-11-24 18:36:30 11月18日 Update

CTOnews.com The training of OpenAI's GPT model requires a large amount of network data, which may involve issues such as data privacy and copyright. To address these issues, OpenAI recently introduced a new feature that allows websites to prevent their web crawlers from scraping data from their websites to train GPT models.

CTOnews.com understands that web crawlers are automated programs that search and retrieve information on the Internet. OpenAI's web crawler, called GPTBot, visits various websites with a certain frequency and saves the content of the web pages for training GPT models.

OpenAI said in its blog post that website operators can prevent GPTBot from scraping data from their websites by blocking access to GPTBot in their Robots.txt file or by blocking their IP addresses. OpenAI also states that "web pages crawled using GPTBot user agents may be used to improve future models and filter out sources that require paid access, are known to collect personally identifiable information (PII), or have text that violates our policies." "For sources that don't meet the exclusion criteria," allowing GPTBot access to your website can help AI models become more accurate and improve their versatility and security. "

However, this does not retroactively remove content previously scraped from websites from ChatGPT's training data.

The Internet provides most of the training data for large language models, such as OpenAI's GPT model and Google's Bard, and acquiring data for AI training has become increasingly controversial. Some websites, including Reddit and Twitter, have taken steps to crack down on AI companies using their user posts for free, while some authors and other creators have filed lawsuits for alleged unauthorized use of their work.

Tags: Website data model training web crawler information content source user web page question interconnection middle table Internet creator can pass address most security Apple Docker Huawei Linux macOS MariaDB Microsoft MySQL NVidia OPPO Reno Apple Shulou Technology MariaDB MySQL macOS