{"id":132,"date":"2024-06-14T08:42:19","date_gmt":"2024-06-14T08:42:19","guid":{"rendered":"https:\/\/www.stanventures.com\/news\/?p=132"},"modified":"2025-10-29T07:31:29","modified_gmt":"2025-10-29T07:31:29","slug":"disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes","status":"publish","type":"post","link":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/","title":{"rendered":"Disallow Crawling of Action URLs in Robots.txt, Says Garry Illyes"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-136\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp\" alt=\"\" width=\"1792\" height=\"1024\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp 1792w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling--300x171.webp 300w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling--1024x585.webp 1024w\" sizes=\"auto, (max-width: 1792px) 100vw, 1792px\" \/><\/p>\n<p>In a recent <a href=\"https:\/\/www.linkedin.com\/posts\/garyillyes_you-should-really-disallow-crawling-of-your-activity-7206189940009791488-WS_M?trk=public_profile\">discussion on LinkedIn<\/a>, Google Analyst Garry Illyes emphasized the importance of disallowing action URLs, such as &#8220;add to cart&#8221; and &#8220;add to wishlist,&#8221; in your robots.txt file.\u00a0<\/p>\n<p>According to Garry, websites that disallow such links from getting crawled sees improved resource allocation while crawling and helps reduce server load.\u00a0<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#what-happened\" >What Happened?\u00a0<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#key-facts-and-figures\" >Key Facts and Figures:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#short-term-and-long-term-impacts\" >Short-term and Long-term Impacts:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#recommendations-from-stan-ventures-on-what-you-should-do-next\" >Recommendations From Stan Ventures on What You Should Do Next:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#tips-and-best-practices\" >Tips and Best Practices<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"what-happened\"><\/span><b>What Happened?<\/b>\u00a0<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-134\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-2.png\" alt=\"\" width=\"512\" height=\"370\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-2.png 512w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-2-300x217.png 300w\" sizes=\"auto, (max-width: 512px) 100vw, 512px\" \/><\/p>\n<p>Garry highlighted that action URLs should be disallowed from being crawled by search engines. These URLs include functions like adding items to a cart or wishlist, which are unnecessary for search indexing.<\/p>\n<p>While Garry says website owners and SEO professionals can implement these recommendations to improve their site&#8217;s efficiency and avoid unnecessary server strain, we have heard from multiple sources that Google has significantly reduced the crawl budget for websites.<\/p>\n<p>Now, it\u2019s clear that Google has been working on reducing unnecessary crawl attempts to make the web more efficient. By encouraging webmasters to disallow non-essential URLs, Google aims to conserve resources and improve the overall performance of its search engine.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-133\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-1.png\" alt=\"\" width=\"512\" height=\"370\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-1.png 512w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/garry-illyes-post-1-300x217.png 300w\" sizes=\"auto, (max-width: 512px) 100vw, 512px\" \/><\/p>\n<p>Garry had earlier <a href=\"https:\/\/www.linkedin.com\/posts\/garyillyes_my-mission-this-year-is-to-figure-out-how-activity-7180832169156562945-aleJ?utm_source=li_share&amp;utm_content=feedcontent&amp;utm_medium=g_dt_web&amp;utm_campaign=copy\">written on LinkedIn<\/a>: \u201cMy mission this year is to figure out how to crawl even less, and have fewer bytes on wire.<\/p>\n<p>A few days ago there was a post on a Reddit community about how, in the OC&#8217;s perception, Google is crawling less than previous years. In the grand scheme of things that&#8217;s just not the case; we&#8217;re crawling roughly as much as before, however scheduling got more intelligent and we&#8217;re focusing more on URLs that are more likely to deserve crawling.<\/p>\n<p>However, we should, in fact, crawl less. We should, for example, be more intelligent about caching and internal cache sharing among user agents, and we should have fewer bytes on wire.<\/p>\n<p>If you&#8217;ve seen an interesting IETF (or other standards body) internet draft that could help with this effort, or an actual standard I might&#8217;ve missed, send it my way. Decreasing crawling without sacrificing crawl-quality would benefit everyone.\u201d<\/p>\n<h2><span class=\"ez-toc-section\" id=\"key-facts-and-figures\"><\/span><b>Key Facts and Figures:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Action URLs like https:\/\/example.com\/product\/scented-candle-v1?add_to_cart can unnecessarily consume server resources when crawled.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Disallowing these URLs can prevent excessive crawling and optimize site performance.<\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"short-term-and-long-term-impacts\"><\/span><b>Short-term and Long-term Impacts:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Short-term:<\/b> Immediate reduction in server load and improved site performance.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Long-term:<\/b> Enhanced user experience and better resource management for website owners.<\/li>\n<\/ul>\n<p>For websites and SEO professionals, managing crawling directives in the robots.txt file can lead to more efficient indexing and better overall site health.<\/p>\n<p>In the past, webmasters have faced issues with excessive crawling that have led to significant server slowdowns and performance issues.<\/p>\n<p>Effectively managing crawler access through robots.txt has proven effective in mitigating such issues.<\/p>\n<p>The current emphasis on disallowing action URLs reflects ongoing efforts to optimize search engine interactions with websites.<\/p>\n<p>In future updates to crawling practices, Google may further refine how it interacts with dynamic and action-oriented content.<\/p>\n<p>Since Webmasters and digital marketers want to mostly align with the recommendations of Google rather than go against it, they are likely to adopt these practices to enhance site performance and user experience.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"recommendations-from-stan-ventures-on-what-you-should-do-next\"><\/span><b>Recommendations From Stan Ventures on What You Should Do Next:<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ol>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Update your robots.txt file:<\/b> Add disallow rules for action URLs.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Monitor crawling activity:<\/b> Use Google Search Console to track crawler behavior.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Optimize site performance:<\/b> Regularly review and update crawling directives.<\/li>\n<\/ol>\n<h2><span class=\"ez-toc-section\" id=\"tips-and-best-practices\"><\/span><b>Tips and Best Practices<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Avoid using &#8220;nofollow&#8221; for internal links to action URLs if robots.txt rules are in place.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Alternatively, Some search engine crawlers primarily follow and index GET requests. By converting action URLs to use POST, you reduce the likelihood that these URLs will be crawled, as many crawlers are less likely to perform POST requests due to their potential to change server state or execute actions.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>In a recent discussion on LinkedIn, Google Analyst Garry Illyes emphasized the importance of disallowing action URLs, such as &#8220;add to cart&#8221; and &#8220;add to wishlist,&#8221; in your robots.txt file.\u00a0 According to Garry, websites that disallow such links from getting crawled sees improved resource allocation while crawling and helps reduce server load.\u00a0 What Happened?\u00a0 Garry [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-132","post","type-post","status-publish","format-standard","hentry","category-seo"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.9 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Disallow Crawling of Action URLs in Robots.txt<\/title>\n<meta name=\"description\" content=\"Learn why it&#039;s crucial to disallow action URLs like &quot;add to cart&quot; in your robots.txt to optimize your website&#039;s performance and conserve server resources.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Disallow Crawling of Action URLs in Robots.txt\" \/>\n<meta property=\"og:description\" content=\"Learn why it&#039;s crucial to disallow action URLs like &quot;add to cart&quot; in your robots.txt to optimize your website&#039;s performance and conserve server resources.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/\" \/>\n<meta property=\"og:site_name\" content=\"Stan Ventures\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/StanVentures\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-06-14T08:42:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-10-29T07:31:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp\" \/>\n<meta name=\"author\" content=\"Dileep Thekkethil\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dthekkethil\" \/>\n<meta name=\"twitter:site\" content=\"@stanventures\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dileep Thekkethil\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/\"},\"author\":{\"name\":\"Dileep Thekkethil\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/person\\\/87d00ff18daf9650e7c925ae4bf86efb\"},\"headline\":\"Disallow Crawling of Action URLs in Robots.txt, Says Garry Illyes\",\"datePublished\":\"2024-06-14T08:42:19+00:00\",\"dateModified\":\"2025-10-29T07:31:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/\"},\"wordCount\":632,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Disallow-Crawling-.webp\",\"articleSection\":[\"SEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/\",\"name\":\"Disallow Crawling of Action URLs in Robots.txt\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Disallow-Crawling-.webp\",\"datePublished\":\"2024-06-14T08:42:19+00:00\",\"dateModified\":\"2025-10-29T07:31:29+00:00\",\"description\":\"Learn why it's crucial to disallow action URLs like \\\"add to cart\\\" in your robots.txt to optimize your website's performance and conserve server resources.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Disallow-Crawling-.webp\",\"contentUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Disallow-Crawling-.webp\",\"width\":1792,\"height\":1024,\"caption\":\"Disallow Crawling\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Disallow Crawling of Action URLs in Robots.txt, Says Garry Illyes\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#website\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\",\"name\":\"Stan Ventures\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\",\"name\":\"Stan Ventures\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Stan-Ventures.webp\",\"contentUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Stan-Ventures.webp\",\"width\":2001,\"height\":801,\"caption\":\"Stan Ventures\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/StanVentures\\\/\",\"https:\\\/\\\/x.com\\\/stanventures\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/person\\\/87d00ff18daf9650e7c925ae4bf86efb\",\"name\":\"Dileep Thekkethil\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"caption\":\"Dileep Thekkethil\"},\"description\":\"Dileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google\u2019s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.\",\"sameAs\":[\"https:\\\/\\\/stanventures.com\\\/news\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/dileep-pradeep-3705aa53\\\/\",\"https:\\\/\\\/x.com\\\/dthekkethil\"],\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/author\\\/admin_7mxgn8tx\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Disallow Crawling of Action URLs in Robots.txt","description":"Learn why it's crucial to disallow action URLs like \"add to cart\" in your robots.txt to optimize your website's performance and conserve server resources.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/","og_locale":"en_US","og_type":"article","og_title":"Disallow Crawling of Action URLs in Robots.txt","og_description":"Learn why it's crucial to disallow action URLs like \"add to cart\" in your robots.txt to optimize your website's performance and conserve server resources.","og_url":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/","og_site_name":"Stan Ventures","article_publisher":"https:\/\/www.facebook.com\/StanVentures\/","article_published_time":"2024-06-14T08:42:19+00:00","article_modified_time":"2025-10-29T07:31:29+00:00","og_image":[{"url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp","type":"","width":"","height":""}],"author":"Dileep Thekkethil","twitter_card":"summary_large_image","twitter_creator":"@dthekkethil","twitter_site":"@stanventures","twitter_misc":{"Written by":"Dileep Thekkethil","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#article","isPartOf":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/"},"author":{"name":"Dileep Thekkethil","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/person\/87d00ff18daf9650e7c925ae4bf86efb"},"headline":"Disallow Crawling of Action URLs in Robots.txt, Says Garry Illyes","datePublished":"2024-06-14T08:42:19+00:00","dateModified":"2025-10-29T07:31:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/"},"wordCount":632,"commentCount":0,"publisher":{"@id":"https:\/\/www.stanventures.com\/news\/#organization"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp","articleSection":["SEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/","url":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/","name":"Disallow Crawling of Action URLs in Robots.txt","isPartOf":{"@id":"https:\/\/www.stanventures.com\/news\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#primaryimage"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp","datePublished":"2024-06-14T08:42:19+00:00","dateModified":"2025-10-29T07:31:29+00:00","description":"Learn why it's crucial to disallow action URLs like \"add to cart\" in your robots.txt to optimize your website's performance and conserve server resources.","breadcrumb":{"@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#primaryimage","url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp","contentUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Disallow-Crawling-.webp","width":1792,"height":1024,"caption":"Disallow Crawling"},{"@type":"BreadcrumbList","@id":"https:\/\/www.stanventures.com\/news\/disallow-crawling-of-action-urls-in-robots-txt-says-garry-illyes-132\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.stanventures.com\/news\/"},{"@type":"ListItem","position":2,"name":"Disallow Crawling of Action URLs in Robots.txt, Says Garry Illyes"}]},{"@type":"WebSite","@id":"https:\/\/www.stanventures.com\/news\/#website","url":"https:\/\/www.stanventures.com\/news\/","name":"Stan Ventures","description":"","publisher":{"@id":"https:\/\/www.stanventures.com\/news\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.stanventures.com\/news\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.stanventures.com\/news\/#organization","name":"Stan Ventures","url":"https:\/\/www.stanventures.com\/news\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/logo\/image\/","url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Stan-Ventures.webp","contentUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Stan-Ventures.webp","width":2001,"height":801,"caption":"Stan Ventures"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/StanVentures\/","https:\/\/x.com\/stanventures"]},{"@type":"Person","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/person\/87d00ff18daf9650e7c925ae4bf86efb","name":"Dileep Thekkethil","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","caption":"Dileep Thekkethil"},"description":"Dileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google\u2019s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.","sameAs":["https:\/\/stanventures.com\/news","https:\/\/www.linkedin.com\/in\/dileep-pradeep-3705aa53\/","https:\/\/x.com\/dthekkethil"],"url":"https:\/\/www.stanventures.com\/news\/author\/admin_7mxgn8tx\/"}]}},"_links":{"self":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/132","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/comments?post=132"}],"version-history":[{"count":1,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/132\/revisions"}],"predecessor-version":[{"id":5528,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/132\/revisions\/5528"}],"wp:attachment":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/media?parent=132"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/categories?post=132"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/tags?post=132"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}