robots.txt compliance is optional

Lines 152 and 153 of nytimes.com/robots.txt explicitly tell the Internet Archive's crawler to stay away. Yet on 27 August 2026 at 04:32 UTC, the Wayback Machine captured the site anyway. Here is what robots.txt actually obliges anyone to do and how to check what yours did.

A weathered boundary wall with a wide gateway that has no gate and no hinges, a large painted sign bolted beside it reading User-agent asterisk and Disallow slash, and cart tracks running straight through the opening

A voluntary convention

Martijn Koster proposed robots.txt in 1994 as a voluntary convention among crawler operators. It wasn't until September 2022 that it became an official IETF standard: RFC 9309, co-authored by Koster and three Google engineers.

The RFC defines a syntax for publishing a request. Enforcement is entirely outside its scope, as is Crawl-delay, which appears nowhere in the document.

RFC 9309 §1, Introduction: "This document specifies the rules originally defined by the 'Robots Exclusion Protocol' that crawlers are requested to honor when accessing URIs. These rules are not a form of access authorization."

Ultimately, robots.txt only works if external software chooses to respect it. Here's an example from a big website.

Example: New York Times (nytimes.com)

The Times blocks both crawlers by name:

User-agent: archive.org_bot
Disallow: /

User-agent: CCBot
Disallow: /

ChatGPT, asked on 28 August 2026 whether the Internet Archive's crawler respects robots.txt: "The relevant crawler user agent is typically associated with ia_archiver."

archive.org_bot is the real Archive crawler user agent. However, it seems many websites still try to block ia_archiver. ia_archiver was Alexa Internet's and has had no operator since 2022. Blocking ia_archiver is of no help.

However, if you query the Wayback Machine (archive.org_bot) for nytimes.com:

curl 'http://archive.org/wayback/available?url=nytimes.com'

It returns a capture timestamped 20260827043230 (04:32 UTC on 27 August 2026).

Common Crawl (CCBot), on the other hand, respects robots.txt. If you run

IDX=https://index.commoncrawl.org/CC-MAIN-2026-34-index
curl "$IDX?url=nytimes.com%2F*&output=json"

you will see that the August 2026 crawl index contains no more than one record for nytimes.com: com,nytimes)/robots.txt, fetched on 18 August (HTTP 200, 3,509 bytes). It stopped before sending any further requests to nytimes.com.

You can try out these commands with your own website.

Two archive bays beneath one posted sheet reading nytimes.com/robots.txt: the bay labelled Internet Archive (Wayback Machine) is packed with bound volumes, the front one stamped 2026-08-27 04:32, while the bay labelled CCBot holds a single thin slip reading robots.txt only

Internet Archive on the robots.txt file

Alexis Rossi, Internet Archive, 17 December 2016: "Some have asked if we ignore URL exclusions expressed in robots.txt files. The answer is a bit complicated. Historically, sometimes yes and sometimes no; but going forward the answer is 'even less so.'"

Rossi noted that the Archive ignored exclusion directives during end-of-term .gov and .mil crawls in 2008, 2012, and 2016. Four months later, Mark Graham wrote that the Archive had "stopped referring to robots.txt files on U.S. government and military web sites for both crawling and displaying web pages (though we respond to removal requests sent to info@archive.org)," adding, "We are now looking to do this more broadly." No follow-up announcement has clarified what came of that.

To remove captured content, you must email them directly. Their help page warns: "We do not make any guarantees beforehand about the outcome of a request." Furthermore, exclusion works forward only. It can hide a page, but it cannot un-capture one. The same behavior as deleting a secret from a git repository.

What to do

Keep using robots.txt. It is low-cost, and well-behaved crawlers respect it. However, view it as a request, not an enforcement mechanism. If content must not be fetched, secure it with authentication, network policies, or unpublish it entirely.

Most teams write robots.txt once and forget it. Have you ever VERIFIED whether a Disallow rule actually stopped unwanted access?

References

IETF — RFC 9309: Robots Exclusion Protocol
Common Crawl — Index Server
Internet Archive — Wayback Machine APIs
Internet Archive Blogs — Robots.txt Files and Archiving .gov and .mil Websites (Alexis Rossi, 17 December 2016)
Internet Archive Blogs — Robots.txt meant for search engines don't work well for web archives (Mark Graham, 17 April 2017)
Internet Archive Help — How do I request to remove something from archive.org?