Why Websites Are Blocking AI Agents — and What It Means for the Internet

01 / The story

Five kinds of 'AI traffic' that are not the same

When a website says it blocks 'AI,' it may mean very different things. A search crawler builds an index so people can find pages. An AI training crawler collects material to train models. An AI search or answer engine retrieves sources to compose answers, often with links. A scraper extracts information—a broad technique, not a synonym for AI or for illegality. And a user-directed agent fetches pages or drives a browser for a specific person's current task. [1, 2]

These categories overlap in practice. OpenAI, for example, documents separate crawlers—GPTBot for training, OAI-SearchBot for search, and user-triggered access—and says search and training preferences are independent. Cloudflare classifies traffic into Search, Agent and Training categories. One crawl can serve more than one allowed purpose, and the categories are functional distinctions, not mutually exclusive legal ones. [1, 2]

02 / The story

Access and permission are different questions

Robots.txt is the web's traditional 'please don't' note. The standard behind it, RFC 9309, is explicit: these are requests that cooperative crawlers are asked to honor—not access authorization. It is neither a password wall nor a copyright license. [5]

Technical enforcement is different. Cloudflare can reject classified traffic at its network edge, which actually stops requests rather than asking nicely. Rate limits constrain how much traffic is allowed without banning everything. Bot detection decides which policy applies; rate limiting decides how much is permitted. Neither establishes ownership or reuse rights. [3, 4, 6, 10]

And licensing is a third layer. Permission to fetch a page is not permission to reproduce it, redistribute it or train a model on it. Conversely, paying for access does not automatically buy every downstream right. The RSL standard, published as version 1.0 in December 2025, explicitly separates machine-readable usage terms, authorization and payment for exactly this reason. [7]

03 / The story

What changed in 2025 and 2026

On July 1, 2025, Cloudflare announced permission-based default blocking of AI crawlers and launched 'pay per crawl' in private beta, using the HTTP 402 'Payment Required' status code to let sites allow, block or charge for access. This was one company's product policy—not an internet-wide ban, and the beta launch does not prove widespread adoption or meaningful publisher revenue. [3, 6]

The system then got more granular. A July 1, 2026 changelog, effective September 15, 2026, documented separate Search, Agent and Training controls. New-domain defaults block Training and Agent traffic on ad-displaying pages while leaving Search allowed—so a simple 'Cloudflare blocks all AI bots by default' headline is now outdated. [2]

On September 15, 2026, Cloudflare launched a 'Disallow AI Training' option and an 'Accountable' designation for mixed-use crawlers, with Apple, Google and Microsoft designated as meeting requirements or committing to specified timelines. That does not mean every operator already enforces every control. Cloudflare's goal of summary-length controls in early 2027 is a roadmap commitment, not an available feature. [4]

04 / The story

Why site owners are doing this

The concerns are economic and legal, and they are related but separate. Cloudflare describes a weakening exchange in which sites provide content but receive fewer visits than under conventional search—treat that as the vendor's and publishers' rationale, not proof that every AI answer destroys a visit. Automated requests also consume real infrastructure resources. [3, 6, 8]

Reddit's lawsuit against Anthropic shows how contested this is. Reddit filed its California complaint on June 4, 2025, alleging breach of contract, unjust enrichment, trespass to chattels, tortious interference and unfair competition—not simply copyright infringement. Anthropic disputed the claims. On March 30, 2026, a federal court sent the dispute back to California state court. That remand was a jurisdictional decision, not a finding that Anthropic unlawfully scraped Reddit, and the federal docket's 'terminated' label does not mean the state case ended. [8, 9, 11, 12]

Meanwhile, licensing is becoming a market. OpenAI and News Corp announced a multiyear agreement on May 22, 2024, covering access to current and archived journalism; the announcement does not disclose a price, so reported valuations should not be presented as company-confirmed. RSL 1.0, published December 10, 2025, defines a machine-readable vocabulary for usage, licensing and payment terms—an industry specification, not legislation, and not evidence that all AI companies accept its terms. [13, 14]

05 / The story

What this means for everyone else

Publishers gain granular control—they can stay discoverable in search while refusing training—but blocking may also reduce their visibility in AI answers, and licensing tends to favor organizations with distinctive content and bargaining power. [1, 2, 3, 4, 13]

Developers should identify their purpose honestly, honor site policies, budget their requests and prefer authorized APIs or feeds. A user-agent label alone is not strong proof of identity, and access policy and permitted reuse are separate checks. [1, 5, 6, 7]

Retailers face a real dilemma: blanket blocking can also obstruct legitimate shopping and comparison agents. Distinguishing catalog discovery from cart operations and account actions is a policy design worth doing deliberately. [2]

And users may face a trade-off between convenient delegated browsing and incomplete coverage. An internet fragmented into paid or selectively accessible sources is a plausible outcome—not an established inevitability. The open web's next decade depends on how these controls, licenses and court cases settle.