{"id":3826,"date":"2025-08-06T15:43:43","date_gmt":"2025-08-06T15:43:43","guid":{"rendered":"https:\/\/www.stanventures.com\/news\/?p=3826"},"modified":"2025-11-05T09:23:55","modified_gmt":"2025-11-05T09:23:55","slug":"cloudflare-vs-perplexity-ai-crawling-debate","status":"publish","type":"post","link":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/","title":{"rendered":"Cloudflare vs. Perplexity: The Battle Over AI Crawling and Robots.txt Explained"},"content":{"rendered":"<p>Cloudflare has officially delisted and blocked Perplexity AI from crawling websites via its infrastructure, citing \u201cstealth crawling\u201d practices and deceptive bot behavior.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3827\" title=\"Cloudflare vs. Perplexity\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif\" alt=\"Cloudflare vs. Perplexity\" width=\"764\" height=\"401\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif 764w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity-300x157.avif 300w\" sizes=\"auto, (max-width: 764px) 100vw, 764px\" \/><\/p>\n<p>According to Cloudflare, the AI assistant service repeatedly violated its Verified Bots policy by ignoring robots.txt directives, rotating IP addresses and spoofing user agents to disguise its crawlers as legitimate human traffic.<\/p>\n<p>That is a serious accusation. But is it entirely accurate? Or is this a larger battle about how the web treats modern AI assistants?<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_83 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\"><\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#what-exactly-happened\" >What Exactly Happened?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#the-alleged-violations\" >The Alleged Violations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#why-it-matters-the-ai-web-crawl-debate\" >Why It Matters: The AI Web Crawl Debate<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#perplexitys-response-%e2%80%9cthis-is-a-misunderstanding%e2%80%9d\" >Perplexity\u2019s Response: \u201cThis is a Misunderstanding\u201d<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#who-owns-access-to-the-open-web\" >Who Owns Access to the Open Web?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#real-world-examples-to-understand-the-situation\" >Real-World Examples to Understand The Situation\u00a0<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#so-who-is-right\" >So, Who is Right?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#a-web-at-a-crossroads\" >A Web at a Crossroads<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"what-exactly-happened\"><\/span><b>What Exactly Happened?<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Cloudflare, one of the most widely used web infrastructure and security providers on the internet, maintains a Verified Bots Program that allows trusted bots to access websites without interference.<\/p>\n<p>To stay verified, these bots must play by the rules primarily, obeying the robots.txt protocol, identifying themselves properly via IP addresses and user agents and avoiding deceptive crawling tactics.<\/p>\n<p>But Perplexity, according to Cloudflare, was not playing fair.<\/p>\n<p>In a blog post, <a href=\"https:\/\/blog.cloudflare.com\/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives\/\">Cloudflare revealed<\/a> it had been receiving complaints from website owners about suspicious bot activity coming from Perplexity.<\/p>\n<p>Following an investigation, Cloudflare found the AI assistant was not only bypassing site restrictions but also actively disguising itself to sneak in undetected.<\/p>\n<p>The result?<a href=\"https:\/\/www.stanventures.com\/news\/perplexity-to-track-all-online-activity-through-its-new-browser-for-targeted-ads-2538\/\"> Perplexity was delisted<\/a> as a verified bot and blocked across Cloudflare\u2019s vast network of protected websites.<\/p>\n<p>Here is the exact statement Cloudflare gave:<\/p>\n<p>\u201cBased on Perplexity\u2019s observed behavior, which is incompatible with [webmaster] preferences, we have de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling.\u201d<\/p>\n<p>But what exactly qualifies as stealth crawling?<\/p>\n<blockquote class=\"twitter-tweet\">\n<p dir=\"ltr\" lang=\"en\">Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites. <a href=\"https:\/\/t.co\/yToVAmwcwn\">https:\/\/t.co\/yToVAmwcwn<\/a><\/p>\n<p>\u2014 Cloudflare (@Cloudflare) <a href=\"https:\/\/twitter.com\/Cloudflare\/status\/1952362105253847100?ref_src=twsrc%5Etfw\">August 4, 2025<\/a><\/p><\/blockquote>\n<p><script async src=\"https:\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>&nbsp;<\/p>\n<h2><span class=\"ez-toc-section\" id=\"the-alleged-violations\"><\/span><b>The Alleged Violations<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Cloudflare accused Perplexity of engaging in several tactics designed to evade detection:<\/p>\n<h3><b>1. Rotating IP Addresses &amp; ASN Switching<\/b><\/h3>\n<p>Perplexity\u2019s official crawlers are expected to use a known range of IP addresses from a specific ASN (Autonomous System Number).<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3828\" title=\"Rotating IP Addresses &amp; ASN Switching in Perplexity\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Rotating-IP-Addresses-ASN-Switching-in-Perplexity.avif\" alt=\"Rotating IP Addresses &amp; ASN Switching in Perplexity\" width=\"1456\" height=\"254\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Rotating-IP-Addresses-ASN-Switching-in-Perplexity.avif 1456w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Rotating-IP-Addresses-ASN-Switching-in-Perplexity-300x52.avif 300w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Rotating-IP-Addresses-ASN-Switching-in-Perplexity-1024x179.avif 1024w\" sizes=\"auto, (max-width: 1456px) 100vw, 1456px\" \/><\/p>\n<p>Instead, <a href=\"https:\/\/www.stanventures.com\/news\/cloudflare-blocks-ai-crawlers-pay-per-crawl-impact-3552\/\">Cloudflare alleges the service<\/a> used a variety of undeclared IPs from unrelated ASNs which make it impossible to trace or block them reliably.<\/p>\n<h3><b>2. User-Agent Spoofing<\/b><\/h3>\n<p>Even more concerning, according to Cloudflare, was that Perplexity\u2019s bots started disguising themselves as ordinary human browsers.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3830\" title=\"User Agent Spoofing in Perplexity\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/User-Agent-Spoofing-in-Perplexity.avif\" alt=\"User Agent Spoofing in Perplexity\" width=\"855\" height=\"453\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/User-Agent-Spoofing-in-Perplexity.avif 855w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/User-Agent-Spoofing-in-Perplexity-300x159.avif 300w\" sizes=\"auto, (max-width: 855px) 100vw, 855px\" \/><\/p>\n<p>One example: the user agent string used was:<\/p>\n<p>Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/124.0.0.0 Safari\/537.36<\/p>\n<p>This string mimics a user browsing with Chrome on a Mac and a deliberate attempt to avoid bot detection filters, Cloudflare says.<\/p>\n<h3><b>3. Ignoring robots.txt<\/b><\/h3>\n<p>Perhaps the biggest violation of trust: ignoring robots.txt.<\/p>\n<p>This is the industry standard file used by websites to instruct bots what they can and can\u2019t crawl.<\/p>\n<p>Perplexity, Cloudflare claims, bypassed these instructions entirely.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"why-it-matters-the-ai-web-crawl-debate\"><\/span><b>Why It Matters: The AI Web Crawl Debate<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-3829\" title=\"The AI Web Crawl Debate\" src=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/The-AI-Web-Crawl-Debate.avif\" alt=\"The AI Web Crawl Debate\" width=\"1600\" height=\"973\" srcset=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/The-AI-Web-Crawl-Debate.avif 1600w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/The-AI-Web-Crawl-Debate-300x182.avif 300w, https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/The-AI-Web-Crawl-Debate-1024x623.avif 1024w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/p>\n<p>Here is where things get more worth understanding.<\/p>\n<p>Perplexity AI is not denying the traffic occurred.<\/p>\n<p>But they are challenging the entire premise of the accusation. Their argument? \u201cWe are not crawling like Google. We are acting on behalf of users.\u201d<\/p>\n<p>Let\u2019s understand that.<\/p>\n<p>Perplexity says its system only fetches web content in real-time, when a user asks a question.<\/p>\n<p>Unlike traditional crawlers (like Googlebot or Bingbot), which preemptively index billions of pages and store that content in massive databases, Perplexity claims it only pulls content once just long enough to summarize it and display an answer.<\/p>\n<p>So, is that a crawler? Or is that more like a digital assistant, doing what you told it to do?<\/p>\n<p>Perplexity compares its approach to Google\u2019s user-triggered fetchers for example, when Google reads a webpage aloud on Android or when it verifies your site with Search Console.<\/p>\n<p>These fetches also bypass robots.txt and they do not store or reuse content.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"perplexitys-response-%e2%80%9cthis-is-a-misunderstanding%e2%80%9d\"><\/span><b>Perplexity\u2019s Response: \u201cThis is a Misunderstanding\u201d<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>In a lengthy technical rebuttal titled \u201c<a href=\"https:\/\/www.perplexity.ai\/hub\/blog\/agents-or-bots-making-sense-of-ai-on-the-open-web\">Agents or Bots<\/a>? Making Sense of AI on the Open Web,\u201d Perplexity laid out its side of the story:<\/p>\n<p>\u201cModern AI assistants work fundamentally differently from traditional web crawling.<\/p>\n<p>When you ask Perplexity a question, the AI does not already have that information sitting in a database.<\/p>\n<p>Instead, it fetches it in real-time and uses it immediately to answer your question.\u201d<\/p>\n<p>Perplexity argues that:<\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Their agents only fetch content in response to user prompts.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">The content is not stored, indexed or used to train models.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Their system behaves like a browser or RSS reader and not like a crawler.<\/li>\n<\/ul>\n<p>They also pointed fingers back at Cloudflare, claiming:<\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Cloudflare misattributed traffic from third-party services (like BrowserBase) to Perplexity.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Their blocking decision was based on fundamentally flawed technical analysis.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">They were possibly used as a scapegoat to generate PR for Cloudflare.<\/li>\n<\/ul>\n<p>In their words:<\/p>\n<p>\u201cWhen you misattribute millions of requests, publish inaccurate diagrams, and misunderstand how AI assistants work, you\u2019ve forfeited any claim to expertise in this space.\u201d<\/p>\n<h2><span class=\"ez-toc-section\" id=\"who-owns-access-to-the-open-web\"><\/span><b>Who Owns Access to the Open Web?<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This debate is not just about Perplexity vs. Cloudflare. It is about the future of information access.<\/p>\n<p>If AI tools are blocked from fetching real-time content, even when requested by users does that mean only the biggest players like Google and Microsoft can crawl the web?<\/p>\n<p>Will smaller, independent AI startups be denied access entirely?<\/p>\n<p>If every AI agent is treated as a rogue bot, what happens to innovation in the open web?<\/p>\n<p>This is the core of Perplexity\u2019s argument: if user-initiated agents are misclassified as bots, we\u2019re closing the door to a more dynamic and personalized web experience.<\/p>\n<p>As one Perplexity engineer put it, \u201cImagine your email client being blocked because it fetched a newsletter.\u201d<\/p>\n<h2><span class=\"ez-toc-section\" id=\"real-world-examples-to-understand-the-situation\"><\/span><b>Real-World Examples to Understand The Situation\u00a0<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Say you are using Perplexity to look up restaurant reviews in your area.<\/p>\n<p>You ask, \u201cWhat are people saying about that new bistro in Brooklyn?\u201d<\/p>\n<p>Perplexity fetches relevant content from recent blog posts, Google reviews and local forums. Within seconds, it summarizes the tone: \u201cMixed reviews praised for ambiance, criticized for wait times.\u201d<\/p>\n<p>That info was fetched on demand and not stored. It means no indexing, no training and just helping you, the user.<\/p>\n<p>Cloudflare would label that behavior as rogue bot activity and unless your tool is on its verified list.<\/p>\n<p>Perplexity says that is the problem that today\u2019s web infrastructure is not built to tell the difference between real-time AI assistants and malicious scrapers.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"so-who-is-right\"><\/span><b>So, Who is Right?<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>That depends on where you stand.<\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Cloudflare<\/b> argues: rules are rules. If you bypass robots.txt, spoof your identity, and crawl from unlisted IPs, you can not be trusted. Their job is to protect website owners and in their eyes, Perplexity broke that trust.<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\"><b>Perplexity<\/b> argues: this is a misunderstanding. Their agents are tools acting on behalf of users, not autonomous bots. Blocking them limits innovation, access and the democratization of real-time information.<\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"a-web-at-a-crossroads\"><\/span><b>A Web at a Crossroads<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>This incident raises deeper questions about the future of the web:<\/p>\n<ul>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Should AI agents follow the same rules as search engine bots?<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Is the robots.txt file enough to govern the modern AI-driven internet?<\/li>\n<li style=\"font-weight: 400;\" aria-level=\"1\">Who gets to decide what constitutes legitimate access?<\/li>\n<\/ul>\n<p>What we are seeing here is not just a tech dispute but a value conflict between open access and gatekeeping, between traditional infrastructure and next-gen intelligence.<\/p>\n<p>One thing is clear: the lines between bots, browsers, assistants and agents are blurring fast. And if infrastructure providers can not keep up, it may not be the bots that suffer\u2014it could be the users.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Cloudflare has officially delisted and blocked Perplexity AI from crawling websites via its infrastructure, citing \u201cstealth crawling\u201d practices and deceptive bot behavior. According to Cloudflare, the AI assistant service repeatedly violated its Verified Bots policy by ignoring robots.txt directives, rotating IP addresses and spoofing user agents to disguise its crawlers as legitimate human traffic. That [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15],"tags":[],"class_list":["post-3826","post","type-post","status-publish","format-standard","hentry","category-ai"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Cloudflare vs Perplexity: AI Crawling Battle Explained<\/title>\n<meta name=\"description\" content=\"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Cloudflare vs Perplexity: AI Crawling Battle Explained\" \/>\n<meta property=\"og:description\" content=\"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/\" \/>\n<meta property=\"og:site_name\" content=\"Stan Ventures\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/StanVentures\/\" \/>\n<meta property=\"article:published_time\" content=\"2025-08-06T15:43:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-11-05T09:23:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif\" \/>\n<meta name=\"author\" content=\"Dileep Thekkethil\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dthekkethil\" \/>\n<meta name=\"twitter:site\" content=\"@stanventures\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Dileep Thekkethil\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/\"},\"author\":{\"name\":\"Dileep Thekkethil\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/person\\\/87d00ff18daf9650e7c925ae4bf86efb\"},\"headline\":\"Cloudflare vs. Perplexity: The Battle Over AI Crawling and Robots.txt Explained\",\"datePublished\":\"2025-08-06T15:43:43+00:00\",\"dateModified\":\"2025-11-05T09:23:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/\"},\"wordCount\":1255,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2025\\\/08\\\/Cloudflare-vs.-Perplexity.avif\",\"articleSection\":[\"AI\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/\",\"name\":\"Cloudflare vs Perplexity: AI Crawling Battle Explained\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2025\\\/08\\\/Cloudflare-vs.-Perplexity.avif\",\"datePublished\":\"2025-08-06T15:43:43+00:00\",\"dateModified\":\"2025-11-05T09:23:55+00:00\",\"description\":\"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2025\\\/08\\\/Cloudflare-vs.-Perplexity.avif\",\"contentUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2025\\\/08\\\/Cloudflare-vs.-Perplexity.avif\",\"width\":764,\"height\":401,\"caption\":\"Cloudflare vs. Perplexity\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/cloudflare-vs-perplexity-ai-crawling-debate-3826\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cloudflare vs. Perplexity: The Battle Over AI Crawling and Robots.txt Explained\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#website\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\",\"name\":\"Stan Ventures\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#organization\",\"name\":\"Stan Ventures\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Stan-Ventures.webp\",\"contentUrl\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/wp-content\\\/uploads\\\/2024\\\/06\\\/Stan-Ventures.webp\",\"width\":2001,\"height\":801,\"caption\":\"Stan Ventures\"},\"image\":{\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/StanVentures\\\/\",\"https:\\\/\\\/x.com\\\/stanventures\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/#\\\/schema\\\/person\\\/87d00ff18daf9650e7c925ae4bf86efb\",\"name\":\"Dileep Thekkethil\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g\",\"caption\":\"Dileep Thekkethil\"},\"description\":\"Dileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google\u2019s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.\",\"sameAs\":[\"https:\\\/\\\/stanventures.com\\\/news\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/dileep-pradeep-3705aa53\\\/\",\"https:\\\/\\\/x.com\\\/dthekkethil\"],\"url\":\"https:\\\/\\\/www.stanventures.com\\\/news\\\/author\\\/admin_7mxgn8tx\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Cloudflare vs Perplexity: AI Crawling Battle Explained","description":"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/","og_locale":"en_US","og_type":"article","og_title":"Cloudflare vs Perplexity: AI Crawling Battle Explained","og_description":"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.","og_url":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/","og_site_name":"Stan Ventures","article_publisher":"https:\/\/www.facebook.com\/StanVentures\/","article_published_time":"2025-08-06T15:43:43+00:00","article_modified_time":"2025-11-05T09:23:55+00:00","og_image":[{"url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif","type":"","width":"","height":""}],"author":"Dileep Thekkethil","twitter_card":"summary_large_image","twitter_creator":"@dthekkethil","twitter_site":"@stanventures","twitter_misc":{"Written by":"Dileep Thekkethil","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#article","isPartOf":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/"},"author":{"name":"Dileep Thekkethil","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/person\/87d00ff18daf9650e7c925ae4bf86efb"},"headline":"Cloudflare vs. Perplexity: The Battle Over AI Crawling and Robots.txt Explained","datePublished":"2025-08-06T15:43:43+00:00","dateModified":"2025-11-05T09:23:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/"},"wordCount":1255,"commentCount":0,"publisher":{"@id":"https:\/\/www.stanventures.com\/news\/#organization"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif","articleSection":["AI"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/","url":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/","name":"Cloudflare vs Perplexity: AI Crawling Battle Explained","isPartOf":{"@id":"https:\/\/www.stanventures.com\/news\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#primaryimage"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#primaryimage"},"thumbnailUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif","datePublished":"2025-08-06T15:43:43+00:00","dateModified":"2025-11-05T09:23:55+00:00","description":"Cloudflare vs Perplexity sparks a web-wide debate on stealth crawling, bots, and how AI agents should follow robots.txt on the internet.","breadcrumb":{"@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#primaryimage","url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif","contentUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2025\/08\/Cloudflare-vs.-Perplexity.avif","width":764,"height":401,"caption":"Cloudflare vs. Perplexity"},{"@type":"BreadcrumbList","@id":"https:\/\/www.stanventures.com\/news\/cloudflare-vs-perplexity-ai-crawling-debate-3826\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.stanventures.com\/news\/"},{"@type":"ListItem","position":2,"name":"Cloudflare vs. Perplexity: The Battle Over AI Crawling and Robots.txt Explained"}]},{"@type":"WebSite","@id":"https:\/\/www.stanventures.com\/news\/#website","url":"https:\/\/www.stanventures.com\/news\/","name":"Stan Ventures","description":"","publisher":{"@id":"https:\/\/www.stanventures.com\/news\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.stanventures.com\/news\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.stanventures.com\/news\/#organization","name":"Stan Ventures","url":"https:\/\/www.stanventures.com\/news\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/logo\/image\/","url":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Stan-Ventures.webp","contentUrl":"https:\/\/www.stanventures.com\/news\/wp-content\/uploads\/2024\/06\/Stan-Ventures.webp","width":2001,"height":801,"caption":"Stan Ventures"},"image":{"@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/StanVentures\/","https:\/\/x.com\/stanventures"]},{"@type":"Person","@id":"https:\/\/www.stanventures.com\/news\/#\/schema\/person\/87d00ff18daf9650e7c925ae4bf86efb","name":"Dileep Thekkethil","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/911bd385b9da54d4a69f19f536a6419e576244371bd6e7d96f06c583dd402fa9?s=96&d=mm&r=g","caption":"Dileep Thekkethil"},"description":"Dileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google\u2019s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.","sameAs":["https:\/\/stanventures.com\/news","https:\/\/www.linkedin.com\/in\/dileep-pradeep-3705aa53\/","https:\/\/x.com\/dthekkethil"],"url":"https:\/\/www.stanventures.com\/news\/author\/admin_7mxgn8tx\/"}]}},"_links":{"self":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/3826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/comments?post=3826"}],"version-history":[{"count":2,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/3826\/revisions"}],"predecessor-version":[{"id":3843,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/posts\/3826\/revisions\/3843"}],"wp:attachment":[{"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/media?parent=3826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/categories?post=3826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.stanventures.com\/news\/wp-json\/wp\/v2\/tags?post=3826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}