In the latest Search Off the Record podcast, Google’s Gary Illyes breaks down the internal data on crawl issues, highlighting how faceted navigation and “action parameters” are silently destroying crawl budgets across the web.
If you think your site’s crawl budget is being spent on your high-value content, think again. According to new internal data from Google, nearly 85% of major crawl issues stem from structural “traps” that waste Googlebot’s resources on useless URLs.
In the latest episode of Google’s Search Off the Record podcast, Search Relations team members Gary Illyes and Martin Splitt reviewed the “2025 Wrapped” of crawl issue reports. The findings reveal that despite years of technical SEO education, websites are still bleeding crawl budget through four primary wounds—with one massive culprit responsible for half of all reports.
Here are the four ways websites are accidentally destroying their own crawl budget, according to Google.
The Budget Black Hole: Faceted Navigation (~50% of Issues)
The single biggest drain on crawl budget remains faceted navigation. Illyes revealed that roughly half of all severe crawling reports come from sites allowing users to filter content (like price, color, or brand) without controlling the resulting URLs.
The Crawl Budget Impact: When a site creates a unique URL for every filter combination, it generates millions of low-value pages. Illyes warned that when Googlebot discovers these vast new URL patterns, it goes into discovery mode.
“Googlebot will want to crawl all those URLs to make a decision whether it should crawl or should not crawl those URLs,” Illyes explained. During this phase, the bot acts “like a madman,” burning through your site’s crawl capacity and potentially crashing the server—leaving zero budget for your actual product or content pages.
The “Action” Trap: Doubling Your URL Space (~25% of Issues)
A surprising 25% of crawl budget waste comes from Action Parameters. These are URLs that perform a function rather than display unique content, such as ?add-to-cart=true, ?wishlist=add, or ?action=save.
The Crawl Budget Impact: By simply having an “Add to Cart” link that uses a GET parameter, a site instantly doubles the number of URLs Googlebot sees. If you add a “Wishlist” parameter, you triple it.
“The things that Googlebot tends not to do is shop around on the internet. It will not buy your weirdo hoodie,” Illyes joked. Since Googlebot doesn’t make purchases, every millisecond it spends crawling an ?add-to-cart URL is a direct waste of budget that could have been used to index a new blog post or product page.
The Legacy Leak: Session IDs (~10% of Issues)
Despite being an outdated practice, Irrelevant Parameters (specifically Session IDs) still account for 10% of crawl issues.
The Crawl Budget Impact: If your site appends a unique ID to the URL for every visitor (e.g., ?sid=12345), Googlebot views every single session as a unique page. This creates a massive amount of near-duplicate content that dilutes the value of the main page and wastes crawl activity on temporary, useless URLs.
The Infinite Loop: Calendar Spaces (~5% of Issues)
The final category involves Infinite Spaces, often caused by calendar widgets or event plugins that allow a user—or a bot—to click “Next Month” forever.
The Crawl Budget Impact: If a calendar widget generates a valid URL for the year 3000, Googlebot may try to crawl it. Illyes noted instances where plugins generated these infinite traps on every single path of a website. This traps the crawler in a loop of empty content, exhausting the budget before it ever reaches valuable pages.
How to Reclaim Your Crawl Budget
The podcast highlighted that while Google attempts to identify these patterns, the responsibility falls on SEOs and developers to block them.
To stop these leaks and focus Googlebot on your money pages, Stan Ventures recommends the following based on Google’s insights:
- Audit Your Logs: Use server logs to see if Googlebot is spending time on ?add-to-cart or filter parameters.
- Use Robots.txt: This is the most effective defense. Disallow crawling of specific parameters (like Disallow: /*?add-to-cart=) and faceted navigation paths.
- Switch to POST Requests: Work with developers to ensure actions like “Add to Cart” use HTTP POST requests (which bots generally don’t follow) rather than GET requests.
Key Takeaway for Webmasters
- Crawl budget isn’t just about speed; it’s about efficiency
- If you aren’t actively blocking these four categories, you are likely asking Google to crawl trash instead of treasure.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.