In a recent episode of Search Off the Record, Martin Splitt and Gary Illyes offered an unfiltered look into how internet standards are created.
The conversation unpacked a process that underpins nearly every online interaction, from how websites are indexed to how data moves across networks.
The episode serves as a detailed guide to how protocols like robots.txt evolve from informal conventions into formalized standards.
It also explains the roles of major standards organizations, the logic behind slow timelines, and the very real consequences of getting it wrong.
What Exactly Is an Internet Standard?
At its core, a standard is an agreement. It ensures that the numerous systems, browsers, servers, and devices that comprise the web communicate using the same language.
According to Splitt and Illyes, standards regulate the structure of websites (HTML), the transfer of data (HTTP, TCP/IP), and search behavior (robots.txt, sitemap.xml).
Standards can start in many ways—an engineer’s idea, an industry workaround, or even a meme.
However, transforming that idea into a stable and trusted specification requires oversight by a standards body.
The most prominent include:
- IETF (Internet Engineering Task Force): Focuses on transport protocols and lower-level internet infrastructure.
- W3C (World Wide Web Consortium): Oversees web-related standards like HTML and CSS.
- WHATWG: Maintains living web standards such as the current HTML spec.
- ECMA: Responsible for ECMAScript, the standard behind JavaScript.
- RSS Advisory Board: Manages standards related to syndication feeds.
The first step in creating a standard is deciding where it belongs. Each body serves a different technical layer of the internet. As Illyes notes, “You’d take TCP replacement ideas to the IETF, not W3C.”
The Journey from Idea to Standard
Standards don’t begin with authority; they begin with consensus. And consensus takes time.
At the IETF, there are typically three paths:
- Informational RFC: Non-binding and minimally reviewed. Used to share ideas, not enforce them.
- Working Group Adoption: A team of subject-matter experts takes on a draft and develops it collectively.
- Independent Submission via Dispatch: An open call for feedback when no clear working group exists.
Once adopted, the draft is subject to exhaustive review. “Every line of the document is examined—technical details, security implications, even grammar,” Illyes says. That’s not bureaucracy; it’s a safeguard. A minor ambiguity in a parser rule could create a security vulnerability.
Key milestones include:
- Consensus within the working group
- Formal Last Call for comments
- Review by multiple IETF directorates (security, editorial, etc.)
- Final approval and publication
A standard can be published as a Proposed Standard (subject to change) or an Internet Standard (stable and foundational).
Case Study: Formalizing Robots.txt
For over 25 years, robots.txt has been a guideline for instructing search engines on which areas of a website to disregard. However, due to the lack of a formal standard, various parsers interpreted the rules inconsistently. That inconsistency created confusion for site owners and potential vulnerabilities for developers.
Google engineers pushed to formalize it through the IETF. The benefits include:
- Unified behavior across platforms
- Improved parser reliability
- Ability to open source Google’s parser for public use
- Simplified guidance for site administrators
This standardization effort made the file more predictable and secure. It’s now a reliable tool that every major search engine interprets the same way.
Why Not Standardize Everything?
The team briefly considered making sitemaps.xml an official standard. But after weighing the cost against the benefit, they decided it wasn’t necessary.
Unlike robots.txt, sitemap.xml has been implemented consistently since its introduction in 2005. It doesn’t suffer from divergent interpretations. Without a clear benefit, such as eliminating ambiguity or resolving security risks, standardizing it would be a costly formality.
This reflects a principle shared throughout the conversation: standards are for solving real problems, not codifying already-settled practices.
The Long Timelines Are a Feature, Not a Bug
One recurring theme in the podcast is the length of time it takes to move from draft to standard, often years. That slowness, the hosts argue, is essential.
Standards must be durable. They often underpin systems that affect billions of people. A rushed or poorly designed specification can introduce security holes or break interoperability.
Take file size limits in robots.txt. During standardization, the team set a 500 KB cap. Why? To prevent buffer overflow attacks and keep parsers from overloading. Without careful review, such risks might have been missed.
Even the wording of a standard is scrutinized. Terms like “MUST,” “SHOULD,” and “MAY” are capitalized for a reason: they define obligations versus recommendations. Misusing one can change implementation behavior across the internet.
Who Can Participate? Almost Anyone
Another standout point is that the process is open. Within the IETF, there are no membership requirements. You can submit a draft, join meetings, and contribute to discussions without needing formal credentials.
Many standards bodies follow similar practices. W3C and WHATWG host discussions publicly, often on GitHub. TC39 (JavaScript’s governing group) also operates transparently. The open nature of these organizations makes them rare examples of global-scale collaboration.
Practical Lessons for Developers and Technologists
If you work in web development, infrastructure, or technical product design, here’s what you can take from this episode:
Respect existing standards: Don’t reinvent solutions. Build on established protocols.
Read the RFCs: They’re not just for academics. They often include practical implementation guidelines.
Get involved: Whether you submit a draft or just monitor discussions, participation is open.
Don’t chase standards for prestige: Only standardize what truly needs clarity, stability, or wide adoption.
Test before proposing: Show that your idea works with a reference implementation.
Key Takeaways
- Standards are built by consensus, not imposed by authority.
- IETF, W3C, ECMA, and others each govern different technical layers.
- Robots.txt became a standard to fix inconsistent behavior across the web.
- Standards take years because reliability and security are paramount.
- Anyone can participate in shaping the future of the internet.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.