I wrote that AI bots ignore llms.txt. Two days later I had to take it back

I put up a file and waited

At the start of October my lab site, pw.migurei.ai, had a /llms.txt, and every page pointed to it with <link rel="describedby" href="/llms.txt">. The file is meant as a plain-text summary of a site for language models. I expected OpenAI, Anthropic and Perplexity to come and read it.

On 14 October I went through the logs from the 1st. Googlebot had fetched the file twice. Bingbot once. A few requests came from Google IP addresses with a browser user agent. Nothing from OpenAI, Anthropic or Perplexity, not even during the test runs my own lab drove through their apps.

So I wrote it down: llms.txt is read by search crawlers, not AI bots.

Two days later it was wrong

On 16 October GPTBot fetched /llms.txt, following the link from the home page, in among 18 other pages. A few hours later ClaudeBot fetched it too, after it had already taken all 17 URLs in the sitemap. Both are training crawlers, and both treated the file as one more URL in a crawl, not as something to read first.

On 18 October a crawler calling itself Amazonbot fetched the file. The next morning it requested two URLs listed inside it, each with ): stuck to the end. Both returned 404. It had read the file as raw text and cut each URL at the next space, so the closing bracket of the Markdown link became part of the address.

I ran one more test, and this time I wrote the prediction down on 17 October, before the pages went live: search crawlers would fetch the .md version of a page, AI crawlers would not. It came out the other way round. ClaudeBot fetched both .md files, roughly 1 hour 12 to 1 hour 16 minutes after the HTML. Googlebot and Bingbot fetched neither. On 21 October I marked the claim refuted.

The part that is actually odd

The bots reading llms.txt are training crawlers. The ones that sit behind live answers, OAI-SearchBot, Claude-SearchBot and PerplexityBot, and the user-triggered fetchers, ChatGPT-User, Claude-User and Perplexity-User, made zero requests for llms.txt or any .md file.

A second test also came back empty. From 16 to 19 October I put a unique URL, listed nowhere else, in the llms.txt on migurei.ai. In 72 hours nobody requested it, bot or otherwise. But I had no sign that any crawler had read that source file in the first place, so the zero says very little.

What to do with your site

1. Do not put llms.txt ahead of getting your HTML pages crawled and indexed. Whatever you want AI bots to read belongs in HTML pages listed in your sitemap.
2. If you already have llms.txt, keep it. It costs little, and some crawlers do read it.
3. Make every URL in the file stand clear of punctuation. Crawlers read it as raw text, and a trailing ):, ） or 。 turns a link into a 404.
4. A .md version only reaches crawlers that already have the HTML, because the only way to find it is the link tag inside that HTML. It is not a new way in.

How sure is this

The original claim is refuted, and I am leaving it public. What replaces it is a lab observation: one site, a few weeks, two .md files. AI search crawlers barely visit pw at all, so their zero cannot tell "does not read llms.txt" apart from "does not come to the site". Apps might also get the content through someone else's index or cache that my logs never see. The Amazonbot requests are matched by user agent only, not yet checked against Amazon's published IP ranges. The full numbers are on the pw research pages.