Listing paths in robots.txt makes them public: RFC 9309 says robots.txt is not access authorization, and most sites we scan have /admin or CMS paths in Disallow.

A file anyone can open

robots.txt is the first thing my scanner reads on any site. It is public by design: any person or program can open /robots.txt and read every line. Many owners use Disallow to keep crawlers away from the admin screen, the CMS login or a test folder. Each of those lines is also a signpost.

What the standard says

RFC 9309, the 2022 standard for robots.txt, says it plainly. Section 1: "These rules are not a form of access authorization." Section 3: "Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable."

Security testers know this. The OWASP testing guide lists robots.txt, sitemaps and the .well-known folder among the files to read first when gathering information about a site.

What I saw

On 22 October 2026 I went through the robots.txt saved from my latest scan. About 10 of the 14 sites had /admin (one scan, 22 October 2026). Most of that was not the owner's choice: it is the default list of Shopify, the same on every shop. A path that is identical on thousands of shops and sits behind a login tells a stranger very little.

One site went further. Its robots.txt named a server configuration file and a file inside its CMS template folder. Those are not defaults. Someone wrote them in to keep crawlers away, and in doing so told everyone where they are.

So the finding is smaller than the headline. Most /admin lines carry little weight. The few hand-written ones are worth a look.

What you can do

1. Open your own robots.txt and read it as a stranger would. Ask of each line: what does this tell someone about my site?
2. Protect admin screens, test areas and config files on the server: login, IP restriction, or simply not serving them. robots.txt only asks polite crawlers to stay away.
3. If a path is private and not a platform default, take it out of robots.txt and close it properly instead.

What this does not prove

• Being named in robots.txt does not mean a path can be opened. My scanner obeys robots.txt and never requests a disallowed path, so I do not know whether any of them was open.
• I do not judge which path is "sensitive". No official list exists, OWASP included, so my report lists the paths and leaves the judgement to you.
• 14 sites, one scan. That is a snapshot, not a survey. The full numbers are on the research page below.

The full numbers, and how this fits with the rest of a site's openness to AI: Open and closed to AI, at https://pw.migurei.ai/en/research/open-and-closed-to-ai.