In a recent episode of Google’s Search Off the Record podcast, Gary Illyes from the Search Relations team reconfirmed what’s been widely believed since 2020 the 1 million-page crawl budget threshold still stands.
But there is a twist that page count alone does not define how often or how deeply Googlebot crawls your website.
We all have assumed that if your website has thousands (or millions) of pages, you automatically become a candidate for crawl budget concerns. But after Google’s latest revelations and revisiting some long-standing assumptions it is possible that we have been looking at the wrong metric all along.
Instead, Illyes made it crystal clear that it is site performance, particularly server speed and content delivery, that ultimately determines crawl activity.
This revelation is a sharp reminder that the crawl budget in 2025 is not about quantity but it is about efficiency. Let’s break down what Gary Illyes really meant.
Watch this 👉
Google Says Database Efficiency Trumps Page Volume
For years, the SEO community has referred to the one million-page mark as the threshold where crawl budget starts to matter. According to Gary Illyes, that reference point remains relevant even in 2025. However, it is not the full story.
“I would say one million is probably okay,” Illyes said during the podcast episode.
The word “probably” might not inspire confidence at first glance but in Google-speak, it tells us something important: the threshold is still used as a baseline not a ceiling. There is no definitive cap that says crawling breaks after 1,000,001 pages. Instead, it implies that around that number, Google may start to assess your site’s performance more critically.
That does not mean that smaller sites are off the hook. Even if your site is well under one million pages, poor server performance or database inefficiencies can result in slower crawling and delayed indexing issues that directly affect visibility in search.
It’s Not About How Big You Are — It’s How Fast You Can Serve
Now here is where the conversation takes a crucial turn. Gary Illyes emphasized that site speed and backend efficiency matter more than just the number of URLs.
“If you are making expensive database calls, that’s going to cost the server a lot,” Illyes noted.
This means that a website with a few hundred thousand pages but plagued with slow database calls, dynamic rendering issues or poor server configurations could suffer more in terms of crawlability than a static site with over a million pages.
This reinforces a central principle we often discuss internally with developers and infrastructure teams: Google does not just care about how many pages you have but it cares how fast it can get them.
Think of it like this: Googlebot is a user. If your server struggles to respond or keeps timing out, you would not blame the user for giving up. So why should Google treat it any differently?
Crawling Is Not the Resource Drain But Indexing Is
This next insight caught even seasoned SEOs off guard. According to Illyes, crawling is not the main concern when it comes to Google’s resource allocation.
“It’s not crawling that is eating up the resources,” he explained. “It’s indexing and potentially serving — or what you are doing with the data when you are processing that data.”
This subtle but important point shifts the SEO narrative. Many of us obsess over crawl stats in Google Search Console or tools like Screaming Frog, believing that more crawling equals more visibility.
But the reality is that Google can crawl a lot; it just does not want to waste time indexing poor or slow-loading content.
If your pages load slowly, are bloated with unnecessary scripts or rely too heavily on JavaScript rendering they become harder to index. It is not the act of crawling that bottlenecks your visibility, it is what comes after.
This insight also speaks volumes for content teams. Even the best content in the world won’t perform well if it takes 10 seconds to load or requires five redirects to display.
A Historical Perspective- From 1994 to Today
To understand the significance of the one million-page reference point, it helps to look back. In 1994, the World Wide Web, one of the first search engines, had indexed just 110,000 pages.
By the time WebCrawler launched later that year, it claimed around 2 million pages. At the time these numbers were staggering.
Fast forward to today’s web and you will find eCommerce platforms, global publications and SaaS providers that easily cross the million-page mark through product variations, user-generated content and multi-language deployments.
What is fascinating is that despite exponential growth in web content, Google’s crawl budget principles have remained consistent.
Why? Because it is not about how much is out there and it is about what is worth crawling efficiently.
The True Bottleneck of Database Latency and Infrastructure
Here is the part where most technical SEOs and DevOps professionals should take note: the real limitations are not in Google’s capacity to crawl, they are in your website’s capacity to serve.
Every inefficient database call, every sluggish API endpoint and every unoptimized backend script chips away at your crawl budget.
When Google detects delays, it adjusts its crawl rate to avoid overwhelming your server essentially giving up on frequent revisits.
That is a problem.
Especially for websites that publish time-sensitive content or rely on frequent updates, such as news portals, stock platforms or event ticketing services. If your infrastructure can not keep up, Google will slow down and your freshness factor in search will suffer.
What SEOs Should Do Right Now
Let us not panic and start preparing. The takeaway here is not that everyone should suddenly re-architect their CMS or drop WordPress for static HTML. But rather, Google’s guidance encourages all of us to think differently.
For sites under one million pages, the advice remains consistent:
- Focus on publishing high-quality, useful content.
- Make sure your site loads fast.
- You likely do not need to worry about the crawl budget at all.
For larger sites, or those approaching the million-page range:
- Audit your database queries for performance.
- Implement proper caching strategies to serve dynamic content faster.
- Monitor server response times and reduce reliance on real-time data where possible.
And across the board:
- Shift your energy toward helping Google process and index content more efficiently.
- Use structured data, canonical tags, and a clean HTML hierarchy to simplify interpretation.
- Think beyond crawling. Think holistic technical SEO.
What This Means for the Future of SEO
If there is one lasting insight from this update, it’s this: the fundamentals still matter.
Despite AI advancements, Core Web Vitals and hundreds of search algorithm changes, Google’s crawl budget guidance has stood the test of time.
What changes however, is the emphasis on underlying performance rather than surface-level metrics.
- For SEO teams, this is a cue to work more closely with engineering and DevOps teams.
- For developers it is a sign to treat query performance and database design as search-impacting factors.
- And for site owners, it is a reminder that scaling your site’s size must go hand-in-hand with scaling its speed.
Efficiency Is the New Size
So let us step back.
The message from Google is not new but it is clearer than ever. In 2025, the crawl budget is less about how much content you have and more about how well your infrastructure performs.
It is not just about building big sites and it is about building fast, scalable and indexable sites. Google does not penalize large websites but it penalizes inefficient ones.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.