Googlebot once again dominated web crawling activity in 2025, generating more traffic than any other crawler as artificial intelligence bots surged across the internet, according to Cloudflare’s newly released 2025 Radar Year in Review.
The Cloudflare report shows that while AI crawlers multiplied rapidly, Google’s crawler continued to outpace all competitors in both scale and reach.
Based on the traffic observed across Cloudflare’s global network, it spans more than 330 cities in over 125 countries.
What Did Cloudflare’s 2025 Report Reveal About Googlebot?
Cloudflare’s data shows that Googlebot remained the single most active crawler on the web throughout 2025. The crawler accounted for more than a quarter of all verified bot traffic observed on Cloudflare’s network during the year.
Googlebot alone generated approximately 4.5% of all HTML request traffic, growing with the combined total of all other AI crawlers, which together accounted for about 4.2%.
This means Google’s crawler generated more web traffic by itself than OpenAI, Anthropic, Meta, and other AI platforms combined.
For the third consecutive year the most active IPv4 address block seen by Cloudflare belonged to Googlebot, supporting Google’s continued dominance in large-scale web crawling for search indexing and AI training.
Why Is Googlebot’s Crawl Volume Still So High?
Cloudflare’s attributes Googlebot’s scale to the sheer number of websites Google indexes and the dual role the crawler now plays.
Googlebot is used both for traditional search indexing and for collecting data to support AI-driven experiences.
The report shows that Googlebot’s crawl volume dwarfed that of other AI crawlers, including those operated by OpenAI, Anthropic, and Meta.
Even as AI platforms increased their crawling activity significantly, none came close to matching Googlebot’s overall footprint.
How Much Did AI Crawling Increase in 2025?
One of the most striking findings in the report is the rapid acceleration of AI-driven crawling activity.
Cloudflare recorded a more than 15-fold year-over-year increase in AI “user action” crawling, a category of bots designed to simulate human browsing behavior.
These bots are often associated with training large language models, retrieving content for AI-generated answers, and powering conversational interfaces.
Their growth reflects the broader expansion of generative AI products throughout 2025.
Why Are Publishers Blocking AI Crawlers More Frequently?
Cloudflare found that AI crawlers were the most frequently fully disallowed user agents in robots.txt files during 2025.
This suggests a growing pushback from publishers concerned about content usage, attribution, and traffic loss.
Many site owners appear increasingly willing to restrict AI bots while continuing to allow traditional search crawlers.
This shift reflects ongoing tension between content creators and AI platforms over how data is collected and monetized.
Which AI Platforms Crawl the Most Compared to Traffic Sent Back?
The report highlights stark differences in crawl-to-refer ratios among major AI and search platforms.
This ratio compares how much content a platform crawls versus how much referral traffic it sends back to websites.
Anthropic recorded the highest crawl-to-refer ratio among major platforms. Early in the year, the ratio peaked near 500,000 to one, meaning Anthropic crawled hundreds of thousands of pages for every single visit it sent back.
After May, the ratio stabilized but remained extremely high, ranging between roughly 25,000 to 100,000 to one.
OpenAI also showed a significant imbalance, with its crawl-to-refer ratio spiking to around 3,700 to one in March before declining later in the year.
Perplexity stood out as an exception. Its ratio remained comparatively low, starting below 100 to one, briefly spiking during a crawl surge in late March, and then settling under 200 to one from September onward.
How Do Traditional Search Engines Compare to AI Platforms?
Search engines displayed very different crawl-to-refer patterns compared to AI platforms.
Microsoft’s ratio hovered between 50 to one and 70 to one, showing a relatively stable balance between crawling and traffic sent back.
Google’s ratio fluctuated significantly during the year.
It rose from just over three to one early in the year to nearly 30 to one by April, fell back to around three to one by mid-July, and then gradually increased again toward the end of 2025.
DuckDuckGo maintained a ratio below one to one for most of the year, meaning it sent more traffic than it crawled, before rising to about 1.5 to one in October and remaining elevated through year-end.
Did Google Maintain Its Search Traffic Dominance in 2025?
Cloudflare’s data shows that Google’s dominance in search traffic remained largely unchanged.
The company continued to deliver nearly 90% of all search engine referral traffic observed on Cloudflare’s network.
Other search engines remained distant followers.
Bing accounted for roughly 3.1% of referrals, followed by Yandex at 2.0%, Baidu at 1.4%, and DuckDuckGo at 1.2%. While there were minor fluctuations throughout the year, the overall ranking remained stable.
How Does This Fit Into Broader Internet Trends in 2025?
The findings on bots and crawling were part of Cloudflare’s broader 2025 Radar Year in Review, which analyzed internet trends from January 1 through December 2, 2025.
Cloudflare reported that global internet traffic grew 19% during the year, with growth accelerating significantly after August.
AI adoption, post-quantum encryption, satellite internet expansion, and record-breaking DDoS attacks were also key themes in the report.
Despite the rise of AI bots, Googlebot’s dominance underscores how central traditional search infrastructure remains, even as the internet adapts to AI-driven systems.
What Does the Surge in AI Crawlers Mean for the Web?
The data illustrates a growing imbalance between how much content AI platforms consume and how much traffic they return.
For publishers, this raises questions about sustainability, attribution, and the long-term economics of content creation.
At the same time, the continued dominance of Googlebot suggests that search indexing remains the most significant driver of large-scale crawling activity.
While AI bots are growing, they have not yet displaced traditional search crawlers in terms of overall volume.
Key Takeaways
- Googlebot accounted for over 25% of all verified bot traffic observed by Cloudflare in 2025.
- Googlebot alone generated more HTML request traffic than all other AI crawlers combined.
- AI “user action” crawling increased more than 15 times year over year.
- AI crawlers were the most frequently blocked bots in robots.txt files.
- Anthropic and OpenAI crawled far more content than they returned as referral traffic.
- Google continued to deliver nearly 90% of all search engine referral traffic.
Dipti Arora
AuthorDipti Arora is a Senior Content Writer with over seven years of experience creating impactful content across Digital Marketing, SEO, technology, and business domains. She has a strong background in managing news verticals and delivering editorial excellence. Dipti has contributed to leading publications such as The Times of India and CEO News, where her research-driven storytelling and ability to simplify complex subjects have consistently stood out. She is passionate about crafting content that informs, engages, and drives meaningful results.