What You Should Walk Away With
- Googlebot’s time on your site is capped. Every fetch it spends on a tracking-parameter URL, a feed, or an internal search page is a fetch your commercial pages don’t get.
- Google can’t tell a URL space is worthless without crawling it first. The trap keeps eating budget long before Google deprioritizes it.
- Open search result pages cost you more than crawl budget. Attackers link junk queries at them at scale until your domain ranks for pharmaceuticals or adult terms, and Google can flag the result as hacked.
- Google’s own Search Relations team now files bug reports against WordPress plugins that generate crawl waste at scale. Your plugin stack can create infinite spaces without you touching a thing.
- Your “Crawled, currently not indexed” report is the cheapest crawl audit you’ll ever run. Ours surfaced nine different flavors of junk in one export.
- Fix it at the source first, block in robots.txt second, and leave redirects and 404s crawlable so Google can flush them out of its systems.
The Math Google Never Puts On A Slide
Google’s Martin Splitt and John Mueller spent all of episode 113 on one question: should you block your own site’s search result pages? Part of their answer is about crawl budget. The other part is nastier. Leave those pages open and someone can point links at them until your domain ranks for pharmaceuticals and adult terms, and Google flags you as hacked.
Here’s the part most site owners skip past in Google’s crawl budget docs: Google describes the web as a nearly infinite space that exceeds its ability to explore and index every URL. That sentence is a budget line hiding in plain sight. Your site gets an allocation of Googlebot’s time, shaped by how much your server can handle and how much Google wants your content.
Google’s official guidance says crawl budget is mainly a concern for sites with a million-plus unique pages. And if your site genuinely has 400 clean URLs, sure, relax. But here’s what that guidance quietly assumes: that you know how many URLs your site has. You probably don’t. Your CMS, your cache plugin, your email platform, and your tracking setup are all generating URLs on your behalf, around the clock, without asking.
A 500-post blog can present Googlebot with 50,000 crawlable addresses once you stack parameters, feeds, API endpoints, and pagination on top. You didn’t build a big site. Your stack built one for you.
The page count where Google says crawl budget “officially” matters. Parameter sprawl gets mid-size sites there faster than they think.
Distinct waste patterns in a single “Crawled, currently not indexed” export from our own Search Console. More on that below.
Every one of those fetches came back with nothing Google could index, and the budget was spent regardless.
So Where Do All These URLs Come From?
None of these are exotic. You almost certainly have at least two of them live on your site right now.
Internal Search Pages
Every query anyone types creates a URL, and Google’s own term for the result is an infinite space. It compounds if your results page links out to related queries, since Mueller described sites where one search page hands Googlebot 10 or 20 more. These pages usually skip your cache too, so every crawl runs a fresh database lookup and a ranking pass on your server.
Faceted Navigation
Filters for color, size, price, and sort order multiply against each other until your 900-product catalog is presenting Googlebot with 60,000 near-identical addresses. Gary Illyes shared in a year-end crawl analysis that faceted navigation made up the largest chunk of crawl issues Google reviewed across the web.
Plugin-Made Calendars
Illyes described a WordPress plugin that keeps injecting bogus URLs, “generating calendar infinite spaces on every single path.” A date archive has no end. Googlebot can crawl your event calendar into the year 3000 if nothing stops it.
Tracking Parameters
UTMs, email tokens, affiliate keys, cache busters. Each combination is a new URL to Googlebot. Your newsletter platform alone can spawn thousands of unique addresses for one blog post.
Feeds And API Endpoints
WordPress hangs a /feed/ off nearly every URL you publish and exposes a full REST API at /wp-json/. Useful for the machines you invited over, pure waste for a crawler that’s supposed to be finding your service pages.
Google Must Crawl It To Judge It
Here’s the mechanism that makes this expensive: Google cannot decide a URL space is low-value without fetching it first. The trap consumes crawls before it can be deprioritized. Your best defense happens on your side, not Google’s.
What We Found In Our Own Search Console
We pulled the “Crawled, currently not indexed” report for stanventures.com in July 2026 and read it line by line. Honestly, we expected a handful of stragglers. What came back was a catalog of every tool we’d ever bolted onto the site, each one quietly manufacturing URLs for Googlebot to chew on. Run the same export on your own site and you’ll probably recognize most of what follows, because these are the same tools sitting on your stack.
The ugliest specimens were email links where tracking tokens had been URL-encoded four and five times over, leaving addresses with strings like %2525253Bamp buried inside them. Nobody sat down and built those. A newsletter template had a broken encoding step, every send multiplied it, and Googlebot fetched the results anyway. If you run campaigns through HubSpot or Mailchimp with an affiliate layer stacked on top, go open one of your own tracked links right now and look at what comes after the question mark.
The timing stung. We were mid-recovery from a serious organic traffic decline, which meant we needed Googlebot back on our commercial pages and refreshed content as often as it would come. Instead, a real slice of its visits went to cache-buster parameters and feed URLs that were never going to rank for anything. That’s how this problem actually shows up for you. Not as a penalty, just as your fixes taking longer to register than they should.
Here’s the full inventory, what created each pattern, and what we did about it. Read it with your own export open.
Notice the last row. That’s the lesson we’d underline twice: robots.txt treats the symptom. The email template was the disease. If your report keeps refilling with fresh junk, something upstream is still generating it, and no Disallow line fixes a broken template.
How To Close The Taps On Your Own Site
Export “Crawled, Currently Not Indexed” And Bucket It
Pull the full report from Search Console and sort it into groups: parameters, feeds, APIs, plugin artifacts, dead pages. You’re not fixing URLs one at a time. You’re finding the machines that make them.
Kill Junk At The Source
Broken email encoding, a calendar plugin you stopped using, an internal search box that links its own results. Fix or remove the generator. This is the step most audits skip because it lives outside the SEO tool.
Block The Deterministic Patterns In Robots.txt
Tracking parameters, cache busters, feeds, API endpoints, internal search paths. Mueller’s advice is to write one rule broad enough to cover the whole pattern instead of a labyrinth of narrow ones, because nobody can maintain the labyrinth six months later. Get the boundary right: disallowing /search/? catches your query URLs without catching a real page like search-for-cheese.php. Test against your own export before deploying.
Leave Redirects And 404s Crawlable
Counterintuitive but important. If old URLs now 301 or 404, Googlebot needs to fetch them to see that and drop them. Block them and they sit in limbo, haunting your reports for years.
Confirm Your Canonicals Are Doing Their Job
Every parameter variant should canonical to the clean URL. Most WordPress SEO plugins handle this by default, but verify it on a live ?utm_source URL rather than assuming.
Re-Export In 60 To 90 Days
Blocked URLs drain out of the report slowly, over months. What you’re watching for is new junk. If fresh patterns appear, you missed a generator. Go back to step two.
Six Ways This Goes Sideways
A blunt robots.txt can do more damage than the junk it’s meant to stop. Two of these come straight from episode 113, and one of them nearly caught us.
Blocking Pages You Want Deindexed
Googlebot can’t see a noindex tag on a page it’s forbidden from fetching. Robots.txt controls crawling, not indexing, and blocked URLs can still get indexed from links alone. Pick one tool per job.
Blocking Rendering Assets
Those versioned JS files cluttering your report look like waste. They’re not. Google needs your scripts and styles to render pages. Blocking wp-includes JS to save budget can cost you rankings on every page that depends on it.
Blocking URLs That Now 404
We almost made this one. After removing our markdown export plugin, the instinct was to disallow all those dead .md.txt URLs immediately. That would have frozen them in Google’s memory forever. Letting Google recrawl the 404s clears them permanently.
Reaching For The Removal Tool
Search Console’s removal tool pulls URLs out of results and does nothing whatsoever to stop crawling. Splitt’s read on it was that you’re hiding the symptom, and the crawl load continues untouched while everyone upstairs believes it’s handled.
Serving 500s To Shed Crawl Load
Tempting when your server is buckling under Googlebot. Mueller’s warning is blunt: Google reads repeated 500s as evidence it’s crawling too hard and throttles back across your entire site, service pages included. If you must return an error on those URLs, a 404 does less damage.
Forgetting Your Other Crawlers
If you welcome AI crawlers like GPTBot and ClaudeBot, your blanket rules apply to them too. Sometimes that’s what you want, since no crawler benefits from parameter junk. But decide it on purpose, per user-agent, not by accident.
The label Google may attach to your site in Search Console when this goes wrong. Mueller described crews that fingerprint common CMSs with unblocked search pages, then fire links at thousands of them carrying pharmaceutical or adult queries alongside a phone number. Your search page renders the query in a heading, your domain starts ranking for it, and Google may take a month to catch on. Blocking it on day one costs you five minutes.
Questions We Get About Crawl Waste
My site has 800 pages. Does any of this apply to me?
Check before you decide. Your site has 800 pages you built. It may present Googlebot with 40,000 URLs your stack built. The “Crawled, currently not indexed” export takes ten minutes and settles the question with data instead of a guess.
Is “Crawled, currently not indexed” a penalty?
No. It means Google fetched the URL and chose not to index it. For junk URLs, that’s the system working. The label is harmless. What hurts is the budget those fetches consumed on the way to nothing.
Should I block my internal search result pages?
Yes, and you get two ways to do it. Robots.txt is the easiest and kills the crawling outright, though a blocked URL can still theoretically surface in results. Noindex is cleaner on that count, but Google keeps crawling those pages in order to see the tag. Mueller called noindex the tidier option and robots.txt the practical one. Either works. Just pick one and apply it site-wide.
Does Google treat open search pages as a quality problem?
Not on its own. Google’s old webmaster guidelines did list search result pages as something to block, and that’s since dropped out of the current search policies. Mueller framed it as an efficiency problem rather than a spam one, saying “you’re being very inefficient.” The quality risk arrives through the back door, when someone else’s terms end up ranking on your domain.
What if my category pages run on the search engine?
Some platforms do this, Blogger being the one Mueller named. Category pages earn their place in the index because they show search engines how your site is organized, so find a URL pattern that separates them from open-ended queries and block only the queries. If no such pattern exists, build real category pages. You’ll write better context on them anyway.
Will fixing crawl waste recover my rankings?
On its own, no, and anyone promising that is selling you something. What it does is remove friction: Googlebot revisits your important pages more often, sees your improvements sooner, and re-evaluates faster. If you’re pushing a recovery, that speed matters.
Want Googlebot Spending Its Time On Pages That Make You Money?
We’ll audit your crawl waste, find the generators, and put Googlebot’s attention back on the pages your agency clients are paying you to rank.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.
