About a week after the launch of OpenAI’s SearchGPT, several leading news publishers have opted to block the startup’s web crawler, OAI-SearchBot. The New York Times and other top news sites block SearchGPT web crawling bots due to concerns about content misuse.
Originality.ai, which monitors such activities, found that 14 out of the top 1,000 publishers have blocked OAI-SearchBot. Among these are Wired, The New Yorker, Vogue, Vanity Fair, and GQ. Jon Gillham, CEO of Originality.ai, expressed confusion over the decision. “I am not sure why any publisher would block it,” he said. “It is traffic that publishers want and need.”
OpenAI’s New Search Engine
OpenAI recently introduced SearchGPT, an AI-powered search engine designed to access real-time information from across the internet. The prototype aims to improve the efficiency of web searches by combining conversational capabilities with real-time data.
The startup has tested the technology with a select group of users and publishers, aiming to streamline the search process. Other tech giants and startups have developed similar AI-enhanced search technologies.
OpenAI’s SearchGPT, unveiled last week, does not use OAI-SearchBot to collect data for training its AI models like GPT-5. Instead, the bot is intended to index information for search results. OpenAI has encouraged website owners to allow the bot to ensure their sites appear in search results. Gillham noted that no major news publishers are known to block Google’s search bot, which operates differently.
Publishers’ Concerns and Trust Issues
The New York Times and other top news sites block SearchGPT web crawling bot as a response to perceived threats from AI-powered search engines. The refusal to allow OAI-SearchBot access may stem from a lack of trust. OpenAI’s other web crawler, GPTbot, collects data for AI model training, and many websites have blocked it for this reason. Publishers may be wary that OAI-SearchBot could also be used to gather content for AI training, despite OpenAI’s assurances.
Another possible reason for blocking the bot could be dissatisfaction with how search results are handled. Modern AI-powered search engines often display summaries rather than directing users to the original content, potentially reducing web traffic for publishers.
The New York Times, a significant player in this stance, has taken legal action against OpenAI and Microsoft. The Times alleges that these companies have used its content without permission to develop competing products.
Charlie Stadtlander, a spokesperson for The New York Times, stated, “The Times does not authorize the use of our works for generative search or AI training purposes without an express written agreement.”
The Reason For Distrust
The recent decision by major news publishers to block OpenAI’s OAI-SearchBot reveals deeper issues related to trust and strategic interests. Publishers like The New York Times and other top sites have chosen to exclude this new bot from crawling their content, a move that raises several important points.
The New York Times and other top news sites block SearchGPT web crawling bot to protect their content from being used for AI training. Firstly, trust appears to be a significant factor here. OpenAI has assured that OAI-SearchBot is designed only for indexing content to provide search results, not for training its AI models. However, many publishers remain skeptical.
This skepticism is rooted in past experiences with OpenAI’s other crawler, GPTbot, which was used for collecting data to train AI models. Publishers fear that allowing OAI-SearchBot access could lead to their content being used in a similar manner, potentially impacting their control over their own material and its use in AI development.
Moreover, publishers are increasingly concerned about the evolving nature of search engines. Traditional search engines direct users to original content, driving traffic to the publishers’ sites. In contrast, AI-powered search engines often display summaries or extracts, which might keep users on the search engine’s platform rather than clicking through to the publisher’s site.
Also Read: OpenAI’s AI Detection Tool May Stay Under Wraps Due to Concerns Over Misuse.
