robots.txt
robots.txt is a small text file that sits at the front door of a website, at a fixed address every automated visitor knows to check: your domain followed by /robots.txt. Inside it are a few lines of instruction aimed not at people but at bots — the crawlers that search engines and AI assistants send out to read the web. The file names who it is speaking to and what it permits: this crawler may go everywhere, that one should stay out of the checkout pages, this whole section is off-limits. It is the web's oldest and simplest way of setting the terms on which machines may enter.
It is worth being clear about what kind of instruction this is. robots.txt is a request, not a lock. Well-behaved crawlers — the ones from the major search and AI companies — read the file and honour it. A badly behaved bot can ignore it entirely, because nothing physically stops it; the file only states a preference and trusts the visitor to respect it. So robots.txt is the right tool for guiding legitimate crawlers, and the wrong tool for security. Anything that truly must stay private needs a real lock, not a polite note on the door.
For a business trying to be found by AI, the failure that matters most is the accidental one. A single overly broad line — a rule meant to hide one folder that instead disallows the whole site, or a leftover instruction from a site that was once under construction — can quietly tell every AI crawler to stay away. The page looks perfect to a human visitor, but the assistants never see it, because they knocked, read the note, and left. Because the file is invisible in normal browsing, this kind of mistake can sit unnoticed for a long time.
The human translation is a note pinned to your shop door for a particular kind of visitor. Written well, it waves the right people in and steers them away from the stockroom. Written carelessly, it can read come back never — and the visitors it turns away are precisely the ones now sending customers your way. It costs nothing to check what your note actually says, and the checking is the whole point: most sites never look, and assume the door is open when it isn't.