Crawling and robots.txt Explained

What robots.txt Is

Robots.txt is a plain text file following the Robots Exclusion Protocol (REP), first proposed in 1994 and formalized as an internet standard (RFC 9309) in 2022. It lives at the root URL of a domain (e.g., https://example.com/robots.txt) and contains directives that tell crawlers which parts of the site they are allowed or disallowed from accessing. The file uses a simple line-based format readable by both humans and machines.

When a well-behaved crawler encounters a new domain, the very first request it makes is for that domain's robots.txt. The crawler parses the file, identifies which rules apply to its user-agent string, and filters its crawl queue accordingly. This check happens once per domain (with periodic re-fetching to detect rule changes), not before every individual page request. The result is a set of path-based rules that the crawler enforces for all subsequent requests to that domain.

Robots.txt is advisory, not enforced by any technical mechanism. Unlike authentication or firewalls, robots.txt does not prevent access. It relies on crawlers voluntarily checking and obeying its directives. All major search engine crawlers (Googlebot, Bingbot, etc.) respect robots.txt. Reputable crawling frameworks like Scrapy check it automatically. Ignoring robots.txt is considered unethical in the crawling community and can lead to legal consequences in some jurisdictions.

Syntax and Directives

A robots.txt file consists of one or more groups, each beginning with a User-agent line that specifies which crawler(s) the following rules apply to. After the User-agent line, one or more Disallow and Allow lines define path-based access rules. The wildcard user-agent (*) matches any crawler not explicitly named elsewhere in the file.

Core Directives

User-agent: Identifies the crawler these rules apply to. Matches against the product token in the crawler's User-Agent HTTP header. The value "*" applies to all crawlers. A file can contain multiple groups with different rules for different crawlers, allowing site owners to give preferential access to specific bots (like search engines) while restricting others.

Disallow: Specifies a URL path prefix that the crawler must not access. The value "/admin/" disallows any URL starting with /admin/. An empty Disallow value (Disallow:) explicitly allows all access, which is different from having no Disallow line at all. Multiple Disallow lines can appear in one group to restrict multiple paths.

Allow: Overrides a Disallow for a more specific path. If you disallow /api/ but want to allow /api/public/, an Allow: /api/public/ line achieves this. Allow directives take precedence when their path is more specific (longer) than a competing Disallow. This directive was not in the original 1994 proposal but is supported by all modern crawlers and is part of RFC 9309.

Crawl-delay: Specifies the minimum number of seconds between consecutive requests to this domain. A value of 10 means the crawler should wait at least 10 seconds between requests. Google does not honor Crawl-delay (preferring Search Console's crawl rate setting), but Bing, Yandex, and most other crawlers do respect it. This directive is not in RFC 9309 but is widely recognized.

Sitemap: Points to the XML sitemap file(s) for the site. This directive is not user-agent specific and applies globally. It helps crawlers discover content that might not be reachable through links alone. Multiple Sitemap lines can list multiple sitemap files or sitemap index files.

Pattern Matching

RFC 9309 standardized wildcard support in path matching. The asterisk (*) matches any sequence of characters within a path. The dollar sign ($) anchors a match to the end of the URL. These patterns enable more precise rules: "Disallow: /*.pdf$" blocks all PDF files regardless of directory, while "Disallow: /search?*q=" blocks search result pages with a query parameter. Without these wildcards, site owners would need exhaustive lists of individual paths to achieve the same effect.

How Crawlers Should Handle robots.txt

Proper robots.txt handling involves several considerations beyond simply parsing the file. First, the HTTP status code of the robots.txt request matters. A 200 response means the file exists and should be parsed normally. A 404 or 410 means no robots.txt exists, and the crawler should assume all paths are allowed. A 5xx error means the server is having problems, and the standard recommendation is to treat all paths as disallowed until the file can be fetched successfully (this prevents accidentally crawling restricted content during server outages).

Caching is important for performance. A crawler should not re-fetch robots.txt before every request to a domain. Standard practice is to cache the parsed rules for 24 hours, or respect the Cache-Control headers returned with the robots.txt response if present. Google refreshes its cache at least once per day for actively crawled domains.

User-agent matching follows specific precedence rules defined in RFC 9309. If the file contains a group matching the crawler's specific user-agent string, those rules apply exclusively. If no specific match exists, the wildcard (*) group applies. If no wildcard group exists, all access is allowed. The matching is case-insensitive and uses the product token (the first word of the User-Agent header, before any version number or URL).

Beyond robots.txt: Other Crawl Directives

Robots.txt is one part of a broader system of crawl directives that website owners use to control bot behavior at different levels of granularity.

Meta robots tags provide page-level control within HTML. The tag <meta name="robots" content="noindex, nofollow"> tells crawlers not to index this specific page (noindex) and not to follow its links to discover new pages (nofollow). These tags override any implicit permission from robots.txt and work at the individual page level rather than the path level.

X-Robots-Tag HTTP headers provide the same directives as meta robots tags but via HTTP response headers. This is useful for non-HTML resources (PDFs, images) where you cannot embed a meta tag. The header X-Robots-Tag: noindex tells crawlers not to index the resource.

Canonical link tags tell crawlers which URL should be considered the authoritative version of a page when multiple URLs serve the same content. The tag <link rel="canonical" href="https://example.com/page"> consolidates crawling and indexing signals to the specified URL, reducing crawl waste on duplicate pages.

Nofollow link attributes tell crawlers not to follow a specific link or transfer authority through it. The attribute rel="nofollow" on an anchor tag signals that the linked page should not be discovered or credited through this particular link. Related values include rel="sponsored" for paid links and rel="ugc" for user-generated content.

Common robots.txt Patterns

Large websites use robots.txt strategically to manage their crawl budget and protect sensitive areas. E-commerce sites typically disallow cart pages, checkout flows, account areas, and internal search results (which generate infinite URL combinations). News sites often disallow archives beyond a certain date range, paywalled content, and print versions of articles. Forums disallow user profile pages, login pages, and thread pages sorted by anything other than the default order.

Some organizations use robots.txt aggressively, blocking all crawlers from their entire site or allowing only specific search engine crawlers. Social media platforms often have extensive robots.txt files running hundreds of lines, carefully controlling which sections each crawler type can access. AI training crawlers (GPTBot, CCBot, Google-Extended) are increasingly being blocked as publishers assert control over how their content is used.

Legal Status of robots.txt

The legal enforceability of robots.txt varies by jurisdiction. In the United States, courts have referenced robots.txt in cases involving unauthorized access, but its violation alone does not automatically constitute a legal claim. The hiQ Labs v. LinkedIn decision focused on whether data was publicly accessible, regardless of robots.txt. However, deliberately ignoring robots.txt can be used as evidence of bad faith in broader legal disputes about unauthorized access or terms of service violations.

In the European Union, robots.txt has been referenced in copyright and database rights cases. The German Federal Court of Justice has considered robots.txt restrictions in cases involving news aggregation and content reuse. Regardless of direct legal enforceability, respecting robots.txt demonstrates good faith and reduces litigation risk significantly.