A large-scale analysis by Hostinger shows website owners making clearer choices about AI access. While many sites are cutting off AI training crawlers, search-focused bots like OpenAI’s OAI-SearchBot are being welcomed, reshaping how content is surfaced and controlled online.
Hostinger has released findings based on anonymized server logs covering more than 66 billion bot requests across over five million websites.
The data was collected across three separate six-day periods and categorized using AI.txt project classifications. Together, the numbers point to a growing divide in how websites treat different types of AI crawlers
Bots used to train large language models are increasingly restricted. At the same time, AI bots that retrieve content in response to user searches are expanding their reach. This change reflects a more deliberate approach by publishers who want control over how their content is used.
Training Bots Are Being Shut Out
The sharpest change appears in the data around AI training crawlers.
OpenAI’s GPTBot, which collects content to train models, saw its average site coverage drop from 84% to just 12% during the study window.
Meta’s ExternalAgent generated the highest request volume among training bots, yet the broader category still experienced consistent declines in coverage.
Hostinger attributes this to more websites actively blocking training crawlers, even when those bots continue to make frequent requests.
This trend mirrors earlier industry research referenced by Hostinger.
Previous studies have shown that a large share of major news publishers now block at least one AI training bot.
Concerns range from uncompensated content use to the cost of handling heavy crawler traffic.
Search and Assistant Bots Tell a Different Story
While training bots are losing access, AI bots tied directly to search and user queries are seeing the opposite trend.
OpenAI’s OAI-SearchBot, which retrieves content for ChatGPT’s search experience, reached an average coverage of 55.67% across the analyzed sites.
Other companies show similar growth.
TikTok’s crawler reached roughly 25.67% coverage, driven by more than 1.4 billion requests. Apple’s crawler reached about 24.33% coverage.
These bots operate in response to individual user actions rather than sweeping data collection. They fetch specific pages to answer questions, which may explain why publishers view them as more acceptable.
Traditional Search Crawlers Remain Stable
Unlike AI-related crawlers, traditional search engine bots showed little change. Googlebot maintained about 72% average coverage with nearly 14.7 billion requests. Bingbot remained close to 57.67% coverage during the same period
This consistency highlights how established search engines still occupy a protected position. Blocking them can directly affect visibility in search results, which makes most publishers hesitant to restrict access.
SEO and Marketing Crawlers Lose Visibility
Hostinger also found declining coverage among SEO and marketing crawlers. Ahrefs maintained the largest footprint at around 60% coverage, but the overall category shrank
Two factors help explain this shift.
Many SEO tools now concentrate on sites actively running optimization campaigns. At the same time, publishers are increasingly blocking crawlers that place heavy demands on server resources.
Earlier data cited by Hostinger showed individual AI bots generating hundreds of millions of requests in a single month, turning bandwidth into a real business concern.
Why Website Owners Are Choosing Sides
Taken together, the data show that publishers are no longer treating all bots the same. Instead of focusing on company names, they are looking at what each crawler actually does.
Training bots collect content to improve AI systems without sending traffic back.
Search and assistant bots, by contrast, can place a publisher’s content directly in front of users through AI-powered search results.
Hostinger suggests this functional difference is driving the shift in access decisions.
What This Means for AI Search Visibility
OpenAI’s own guidance aligns with this trend.
The company advises site owners to allow OAI-SearchBot if they want their pages to appear in ChatGPT search results, even if they choose to block GPTBot.
Hostinger points to OpenAI documentation , OAI-SearchBot respects robots.txt rules and controls inclusion in search features, while other user-initiated browsing behaviors follow different technical paths.
Hostinger recommends that publishers review their server logs regularly to understand which bots are accessing their sites. From there, they can make decisions that match their priorities, whether that means limiting server strain, protecting content, or staying visible in AI-driven discovery tools.
What Website Owners Can Do Right Now
Site owners responding to these changes can take a few practical actions:
- Review server logs to identify AI training and search crawlers.
- Block training bots if content reuse or traffic load is a concern.
- Allow search-focused AI bots that support discovery.
- Use CDN-level controls to manage high-volume crawlers.
- Revisit crawler rules as AI search continues to evolve.
Key Takeaways
- AI training crawlers are being blocked far more often than before.
- OpenAI’s search crawler now reaches over half of analyzed websites.
- Traditional search bots like Googlebot remain largely unaffected.
- SEO and marketing crawlers are losing access due to cost and focus.
- Publishers are deciding access based on purpose rather than brand.
Zulekha
AuthorZulekha is an emerging leader in the content marketing industry from India. She began her career in 2019 as a freelancer and, with over five years of experience, has made a significant impact in content writing. Recognized for her innovative approaches, deep knowledge of SEO, and exceptional storytelling skills, she continues to set new standards in the field. Her keen interest in news and current events, which started during an internship with The New Indian Express, further enriches her content. As an author and continuous learner, she has transformed numerous websites and digital marketing companies with customized content writing and marketing strategies.