Google continues to deliver the lion’s share of traffic to publishers, sending 831 times more visitors than AI systems. But behind the numbers lies a growing crisis. Bots are consuming resources, scraping content, and reshaping how information is accessed, without giving publishers much in return.
TollBit’s Q2 2025 State of the Bots report paints a stark picture. Google still accounts for 84.1% of publisher referrals, while AI platforms barely register at 0.102%.
For every visitor an AI tool sends, Google delivers 831. Yet despite this dominance, the balance between discovery and extraction is shifting in troubling ways.
Human visits to publisher websites fell 9.4% between Q1 and Q2 of this year.
Over the same period, AI bot traffic quadrupled, climbing from one in 200 visitors at the start of 2025 to one in 50 by midyear.
That means a significant portion of “visitors” are no longer readers at all but automated systems posing as people.
Why AI Traffic Looks Human
Modern AI bots don’t resemble nnyesterday’s search crawlers. Headless browsers and agent-driven tools mimic human behavior so closely that server logs often can’t tell the difference.
Perplexity’s Comet, Google’s Project Mariner, and other AI browsers open pages, trigger ads, and even solve CAPTCHAs. To a publisher, they appear as real users, but they don’t subscribe, click ads, or engage with content.
This creates a one-sided exchange.
Publishers pay for infrastructure to handle the traffic, but there is no corresponding revenue. Instead of visitors who might convert into subscribers or customers, bots drain resources while redirecting value elsewhere.
Google’s Mixed Role
Even as AI bots gain traction, Google remains publishers’ most important source of readers. Its referral power dwarfs that of every AI system combined. But the value exchange is eroding.
Since the rollout of AI Overviews in late 2024, Googlebot’s crawling has increased 34.8%. At the same time, the number of referrals per crawl fell 24.4%. In plain terms, Google is extracting more content while delivering fewer readers.
This puts publishers in a bind. Blocking Google risks cutting off their largest traffic stream, but allowing heavier crawls without proportional returns raises costs and dependency.
Content Categories Hit Hardest
Some sectors are under greater strain than others.
Professional and B2B sites face the heaviest scraping relative to human visitors. Parenting content saw a staggering 333% increase in AI requests, while deals and shopping jumped 111%.
National news outlets were targeted by five times more real-time retrieval scrapes than training bots, showing the hunger for fresh information.
Regional trends add another wrinkle. Asia-Pacific sites endured triple the number of AI requests compared to U.S. publishers, while European sites saw 27% fewer.
The pattern suggests that looser blocking in certain regions is making them prime targets for AI systems.
How Publishers Are Fighting Back
The pushback is intensifying. The number of publishers blocking AI bots grew 336% year-over-year.
TollBit’s Bot Paywall saw hits surge 360% in just one quarter. Still, compliance remains uneven.
Over 13% of AI bots ignored robots.txt rules in Q2 2025, up from just 3.3% six months prior.
Some AI companies have improved technical behavior.
Anthropic’s Claude, once plagued by errors, cut its failure rate to 4.8% after gaining live web access.
Meanwhile, OpenAI bots triggered more 404 errors by fabricating URLs, a problem that wastes bandwidth and frustrates publishers.
Business Models at Risk
At stake is nothing less than the economic foundation of the web.
Publishers rely on real readers for ad impressions, subscriptions, and engagement. AI tools disrupt this by summarizing and delivering information directly, reducing the incentive for users to click through.
Licensing deals are emerging as a stopgap. OpenAI has signed nearly 40 partnerships, promising better visibility and higher referral rates.
Yet those deals come with tradeoffs: participating publishers also reported 88% more scraping. Whether this balances out in the long term remains uncertain.
Regulation and the Road Ahead
TollBit argues the next step is regulation. If bots can perfectly impersonate human visitors, traffic analytics and revenue models collapse.
Clear rules requiring bots to self-identify would restore transparency and allow publishers to respond on their own terms.
Some experts envision a parallel “agent web” designed specifically for automated systems. This would separate human-facing content from bot-facing feeds, reducing server strain and clarifying value exchanges.
Whether such a system gains traction depends on policy, industry adoption, and pressure from publishers.
What Publishers Can Do Now
Here are some steps publishers can take as they face rising bot traffic and shifting referral dynamics:
- Invest in detection tools. Sophisticated anti-bot systems can filter out disguised AI browsers.
- Experiment with licensing. Partnerships with AI companies may provide visibility but need careful evaluation.
- Reduce reliance on Google. Building direct audience relationships helps buffer against platform dependency.
- Test monetization walls. Bot paywalls could evolve into a new income stream if widely implemented.
- Join collective advocacy. Regulatory change will likely require joint pressure from publishers.
Key Takeaways
- Google delivers 831 times more traffic than AI systems.
- Human visits are declining while AI bots have quadrupled in 2025.
- AI scraping is rising across content categories but generates negligible referrals.
- Publishers are responding with blocking, bot paywalls, and licensing deals.
- Regulatory standards may be needed to sustain online publishing economics.
Zulekha
AuthorZulekha is an emerging leader in the content marketing industry from India. She began her career in 2019 as a freelancer and, with over five years of experience, has made a significant impact in content writing. Recognized for her innovative approaches, deep knowledge of SEO, and exceptional storytelling skills, she continues to set new standards in the field. Her keen interest in news and current events, which started during an internship with The New Indian Express, further enriches her content. As an author and continuous learner, she has transformed numerous websites and digital marketing companies with customized content writing and marketing strategies.