OpenAI has quietly rolled out a significant update to its ChatGPT web crawler, OAI-SearchBot, altering how it is described and what roles it officially performs.
The changes, first spotted by Pieter Serraris, remove references to link gathering and training foundation models while adding new language about data sharing and user-initiated bot behavior.
This change may appear subtle, but its implications for publishers, SEOs, and digital platforms are important.
If OpenAI alters how its bots respect robots.txt rules, or how different crawl agents share information, what does that mean for the future of AI-driven search?
What Exactly Changed in OpenAI’s OAI-SearchBot Description?
The adjustments were first noticed by Pieter Serraris, who shared screenshots and analysis on LinkedIn.
According to him, OpenAI made “big changes in the description of the roles of their crawlers and user agents.” The modifications include both the removal and addition of specific language, indicating a shift in how OpenAI wants the crawler to be understood publicly.
Previously, OpenAI stated that OAI-SearchBot was used for discovering navigational links shown in ChatGPT responses and for helping train OpenAI’s generative AI models.
These references have reportedly been removed, replaced with new clarifications that better align with OpenAI’s updated policies.
Notably, the updated description now frames OAI-SearchBot as part of a broader ecosystem of bots that work more collaboratively and selectively, rather than as an independently significant crawler contributing raw training data.
The visual comparison shared by Serraris highlighted words in red (removed) and green (added), illustrating the extent of the rewrite.
While OpenAI has not issued an official announcement, the documentation changes alone point to a meaningful shift.
Why Is the Removal of “Links” and “Training Data” Significant?
One of the most important deletions is the claim that OAI-SearchBot contributes data used for training foundation models. This matters because:
- It distances the crawler from the sensitive topic of AI companies collecting training data from the open web.
- It positions OAI-SearchBot more as a functional tool serving user-initiated actions rather than a silent large-scale collector feeding model development.
Given ongoing legal scrutiny and publisher pushback over AI model training practices, OpenAI may be seeking to avoid further controversy.
Removing this language does not necessarily mean the bot never played such a role but it signals what OpenAI wants its current and future posture to be perceived as.
Similarly, removing references to collecting navigational links for ChatGPT responses suggests that the bot is no longer directly responsible for link discovery.
This may reflect architectural changes inside ChatGPT or shifts toward using alternative datasets.
Is ChatGPT Now Ignoring Robots.txt for Certain Bot Types?
Perhaps the most striking change highlighted by Serraris is that ChatGPT User, one of OpenAI’s user-agent identifiers, appears to no longer commit to honoring standard robots.txt rules.
OpenAI changed the wording in the documentation from: “The following robots.txt tags apply to all 3 user agents”
to instead reference only OAI-SearchBot and GPTBot.
This implies that ChatGPT User, which acts on behalf of a user performing live requests, may not obey robots.txt directives in the same way true crawlers do.
If accurate, this could raise concerns among publishers who rely on robots.txt to control automated access.
However, there is a nuance: ChatGPT User is not a traditional crawler. It visits sites only in response to specific user-initiated queries, meaning it functions similarly to a browser.
Browsers, not being crawlers, typically do not follow robots.txt. OpenAI appears to be clarifying this behavior, though the change still may unsettle some web administrators.
How Are OAI-SearchBot and GPTBot Now Sharing Information?
The updated description also adds a new statement:
“If your site has allowed both bots, we may use the results from just one crawl for both use cases to avoid duplicate crawling.”
This confirms what many site owners have already observed, increased activity from OpenAI’s bots across multiple user agents. According to Serraris, some logs now show two or even all three OpenAI bots visiting the same site in response to certain prompts.
OpenAI’s new language suggests a desire to reduce redundant crawling and consolidate results among its bot family.
This also reveals a deeper level of internal coordination between OAI-SearchBot and GPTBot, potentially indicating a shared backend or merged processing pipeline.
Why Does It Matter That ChatGPT User Is Now Linked to Custom GPTs and Actions?
Another notable addition is the statement that ChatGPT User is used for:
- Custom GPT requests
- GPT Actions
This means the bot may now visit websites based on workflows or automations users create inside the ChatGPT ecosystem.
In other words, if a user configures a Custom GPT to fetch data from the web, ChatGPT User becomes the agent responsible for these visits.
This significantly expands the number of potential touchpoints OpenAI’s user-agent might have with websites.
It also blurs the lines between browsing, automation, crawling, and scraping, all topics that are actively debated in discussions around AI ethics and data usage.
What Does This Mean for Website Owners Concerned About Crawling?
For many publishers, the biggest concern is not the traffic OpenAI sends because it sends very little but the volume of bot visits compared to the minimal referral value.
As Serraris noted: “For most people, OpenAI crawls your site way more than it sends you any traffic.”
This imbalance is at the heart of current tension between AI companies and content creators. While search engines like Google historically exchange crawling for traffic, AI models often provide summaries without delivering visitors.
OpenAI’s new description improvements do little to change this dynamic.
In fact, the updates may signal that OpenAI is preparing for even more complex web interactions as ChatGPT grows into a platform where multi-step actions and agent-driven workflows become normal.
What Are the Broader Implications of These Changes?
These updates raise several broader questions:
- Is OpenAI shifting to a more user-driven browsing model instead of mass crawling for training?
- Will routers and admins need new tools to manage multiple types of AI agents?
- How might regulations evolve to distinguish between crawlers and “user action bots”?
- Is OpenAI pre-emptively adjusting documentation ahead of future transparency requirements?
The lack of a public announcement suggests OpenAI prefers a low-profile update rather than a public discussion, a strategy common among major AI developers managing sensitive topics around data sourcing.
Key Takeaway
- OpenAI removed mentions of OAI-SearchBot being used for links or model training.
- ChatGPT User bot now doesn’t follow robots.txt rules.
- OAI-SearchBot and GPTBot can share crawl results to avoid duplicates.
- ChatGPT User is now tied to Custom GPTs and GPT Actions.
- OpenAI bots still crawl far more than they send traffic.
Dipti Arora
AuthorDipti Arora is a Senior Content Writer with over seven years of experience creating impactful content across Digital Marketing, SEO, technology, and business domains. She has a strong background in managing news verticals and delivering editorial excellence. Dipti has contributed to leading publications such as The Times of India and CEO News, where her research-driven storytelling and ability to simplify complex subjects have consistently stood out. She is passionate about crafting content that informs, engages, and drives meaningful results.