A new system called Really Simple Licensing (RSL) lets websites set rules and prices for AI companies that scrape their content. Backed by Reddit, Yahoo, Medium, Quora, wikiHow and others, it could shift bargaining power away from model builders and toward publishers.
AI firms have been feeding on the open web for years, pulling in articles, posts, images and guides to train their models. The expectation has been that if it’s online, it’s fair game. That assumption is now being challenged.
This week, a coalition of publishers introduced Really Simple Licensing (RSL), an open content licensing standard meant to set rules for how automated crawlers handle web content.
The People Behind the Idea
The RSL Collective, the nonprofit behind the standard, is led by Eckart Walther, who played a role in developing RSS, and Doug Leeds, a former Ask.com executive. Both men understand the power of small technical signals on the web.
RSS changed the way news and blogs are distributed. Robots.txt gave publishers a simple way to tell search engines what they could or couldn’t index.
RSL builds on those traditions. Instead of saying “don’t crawl this page,” a publisher can now say “crawl it under these conditions.” Those conditions can include attribution, subscriptions, or usage-based fees. It takes what was once a one-line instruction and turns it into a negotiable contract.
I’ve seen how those little text files often get ignored or forgotten, buried at the root of a site. What strikes me about RSL is how it takes something so humble and tries to turn it into the foundation for an entirely new economic relationship.
The Coalition Behind the Push
At launch, the standard has the backing of some of the web’s busiest platforms: Reddit, Yahoo, Medium, Quora, O’Reilly, wikiHow, and Ziff Davis. These companies generate vast amounts of text, images and community knowledge that have become staples in AI training sets.
Their endorsement is, in fact, strategic. If the standard spreads widely, AI builders will find it harder to claim ignorance or sidestep negotiations. Negotiating with one or two publishers is manageable. But negotiating with dozens or hundreds under the same standard is another matter entirely.
How Enforcement Could Actually Work
The RSL tags themselves are just lines of code. On their own, they cannot block a crawler that decides to ignore the rules. That is why the RSL Collective is working with infrastructure companies to enforce compliance.
The best part is that one of the largest content delivery networks has already joined.
Here’s how it would work. When a bot approaches a site, the CDN checks whether the crawler has agreed to the licensing terms. If it has, access is granted. If not, the request can be blocked before it even reaches the site’s server.
That step could make the difference between a polite request and a functioning economic system.
Without enforcement at the network level, RSL would be a symbolic gesture. With it, AI companies face a real cost if they try to bypass the rules.
Legal Fog and Familiar Patterns
The push for RSL comes against the backdrop of lawsuits over AI scraping.
Writers have filed lawsuits claiming their books were copied wholesale into training sets. Getty Images, a leading stock photo provider, has accused AI companies of using millions of photos without clearance. Newsrooms argue that their reporting, often costly and labor-intensive to produce, is being repackaged by systems that never asked and never paid.
Some cases have led to settlements, such as agreements with Anthropic and others, but the courts have yet to issue a definitive ruling on whether scraping for training constitutes fair use or copyright infringement.
The Stakes for AI Developers
For companies training models, the emergence of Really Simple Licensing means a potential new cost structure.
Building large models is already expensive. Adding recurring fees for access to high-quality data could raise the barrier to entry even higher.
Some will likely pay for licenses, calculating that accuracy and credibility are worth the price. Others may retreat to datasets that are in the public domain or already cleared for training. That choice will shape what different AI models can and cannot do.
There is also a technical wrinkle.
A per-crawl fee is relatively simple. A pay-per-inference fee, triggered when a model generates an answer using licensed material, requires detailed logging inside the AI system. Few, if any, companies have built such traceability into their models. Doing so would require engineering work that many have resisted.
Steps for Creators and Publishers
For anyone running a site or producing content, here are some sensible next moves:
- Look at the RSL specification and see if joining the Collective aligns with your goals. Membership is open and free.
- Decide on your licensing stance. Do you want attribution, subscription revenue, or per-use fees? Make your policy clear.
- Talk to your hosting or CDN provider. Check whether they support automated enforcement of RSL tags.
- If you are a small operation, consider collective membership to avoid carrying legal and technical costs alone.
- For AI builders, start exploring ways to log and audit how your models draw on different data sources. Transparency will matter more if usage-based fees take hold.
What Happens Next
The success of RSL will depend on three early signals. First, whether more large publishers adopt it. Second, whether CDNs and hosting providers push enforcement. And third, whether at least one major AI firm agrees to license terms through the system.
If those pieces fall into place, RSL could rapidly become part of the web’s infrastructure. If they don’t, it risks being remembered as a noble attempt that never gained traction.
Either way, it has become far too evident that publishers are done giving away their content without conditions.
Key Takeaways
- RSL builds on robots.txt but adds licensing instructions.
- The RSL Collective is a nonprofit aiming to manage rights at scale.
- Fastly is the first infrastructure partner to back enforcement.
- Legal battles over scraping remain unresolved, raising the stakes.
- For AI developers, RSL could add new recurring costs and force more selective data use.
Dileep Thekkethil
AuthorDileep Thekkethil is the Director of Marketing at Stan Ventures, where he applies over 15 years of SEO and digital marketing expertise to drive growth and authority. A former journalist with six years of experience, he combines strategic storytelling with technical know-how to help brands navigate the shift toward AI-driven search and generative engines. Dileep is a strong advocate for Google’s EEAT standards, regularly sharing real-world use cases and scenarios to demystify complex marketing trends. He is an avid gardener of tropical fruits, a motor enthusiast, and a dedicated caretaker of his pair of cockatiels.