A striking new study reveals that AI-driven bots now make up 28% of Googlebot’s total traffic, signaling a dramatic shift in the web crawling ecosystem.
With OpenAI’s GPTBot and Anthropic’s Claude leading the charge, the rules of search engine optimization (SEO) are being rewritten.
Here’s what’s happening and how it’ll affect your website.
The Rise of AI Crawlers
A report from Vercel unveils the growing dominance of AI bots in web crawling. These bots, including OpenAI’s GPTBot and Anthropic’s Claude, collectively processed close to one billion monthly requests.
Compared to the 4.5 billion fetches by Googlebot, their contribution—28%—is reshaping the digital landscape. GPTBot topped the list with 569 million requests, followed by Claude with 370 million. AppleBot and PerplexityBot added 314 million and 24.4 million fetches, respectively.
This surge highlights the increasing influence of AI bots in shaping how websites are indexed and ranked.
What Makes AI Crawlers Different
Unlike traditional search engines, AI crawlers come with unique behaviors. They lack the ability to execute JavaScript, which limits their capacity to index client-side rendered content. Despite this, they can still retrieve JavaScript files for processing.
Another stark difference lies in their inefficiency—GPTBot and Claude spend over 34% of their efforts crawling 404 error pages, indicating room for improvement.
AI bots also exhibit distinct preferences for content types. ChatGPT, for instance, focuses primarily on HTML, which constitutes 57.7% of its fetches. Claude, in contrast, dedicates over 35% of its requests to images. These behavioral nuances suggest that traditional SEO strategies, centered around full-spectrum crawling, require adjustments to accommodate these new players.
A U.S.-Focused Presence
As per the report, AI bots are currently U.S.-centric. GPTBot operates from Des Moines, Iowa, and Phoenix, Arizona, while Claude is based in Columbus, Ohio. This limited geographic reach underscores the early stages of AI crawler development, especially compared to global search engines.
Adapting to AI Crawlers
Instead of solely catering to search engine crawlers, we must also address the specific requirements of AI bots. Here’s how:
Server-Side Rendering: AI bots’ inability to execute JavaScript means that server-side rendering is essential. By delivering pre-rendered HTML, websites can ensure that critical content is visible and accessible.
Content Structuring: Structured HTML with clear header hierarchies is crucial. Optimize images with detailed alt text and metadata to align with Claude’s image-centric crawling preferences.
Resolving Technical Issues: The high 404 error rates observed among AI bots highlight the importance of keeping sitemaps updated, redirect chains seamless, and URLs consistent. Regular audits to minimize such errors will enhance crawl efficiency.
What Website Owners Should Do Now
The rise of AI crawlers presents both challenges and opportunities. Website owners must take proactive measures to adapt and thrive in this changing ecosystem.
Monitor Bot Behavior: Analytics tools can provide valuable insights into how AI crawlers interact with your website.
Fine-Tune Robots.txt: Use this file to define access permissions for AI bots clearly.
Focus on Accessibility: Ensure that essential content is visible without relying on JavaScript.
Stay Updated: Keep up with emerging trends in AI crawling to refine your optimization strategies.
Key Takeaways
- AI bots now account for 28% of Googlebot’s traffic, signaling a major shift in SEO.
- GPTBot and Claude lead the AI crawler wave, with nearly 1 billion monthly requests.
- Traditional SEO must adapt to AI bots’ behaviors, such as skipping JavaScript.
- Server-side rendering and semantic HTML are critical for optimization.
- AI bots will play an even larger role in the future of web crawling and content indexing.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.