https://emsenn.net/tags/crawling/ Recent content in Crawling on emsenn.net Hugo en-us Tue, 03 Mar 2026 00:00:00 +0000 https://emsenn.net/library/domains/engineering/domains/tech/domains/computing/domains/internet/robots-txt/ Tue, 03 Mar 2026 00:00:00 +0000 https://emsenn.net/library/domains/engineering/domains/tech/domains/computing/domains/internet/robots-txt/ <p><code>robots.txt</code> is a plain-text file placed at the root of a website (e.g., <code>https://example.com/robots.txt</code>) that tells web crawlers which parts of the site they may or may not access. It follows the Robots Exclusion Protocol, first proposed by Martijn Koster in 1994 and codified as an internet standard in RFC 9309 (2022).</p> <h2 id="how-it-works">How it works</h2> <p>A <code>robots.txt</code> file consists of one or more records, each specifying a user-agent (the crawler&rsquo;s identifier) and a set of <code>Allow</code> and <code>Disallow</code> directives. Crawlers are expected to fetch this file before crawling any other page and to respect its directives, though compliance is voluntary — <code>robots.txt</code> is a convention, not an access control mechanism.</p>