On a site OAI-SearchBot is crawling, new pages it fetches for the first time arrive mainly via links from a page it has just fetched; the sitemap and ChatGPT-User are secondary paths.

What I wanted to know

When a new page goes up, how does OAI-SearchBot find it? OpenAI's bot page only says to allow OAI-SearchBot in robots.txt. It says nothing about sitemaps or about submitting URLs.

A site with history

I could read the server logs of an older Vietnamese-language site, covering about four months of 2026. OAI-SearchBot crawls it regularly. In that period the bot fetched close to two thirds of the site's new pages.

For 92% of those first fetches, the request carried a referer: another page on the same site that the bot had fetched a median of 16 seconds earlier. About 6% came within a minute after a ChatGPT user had opened the page. The rest had neither.

Referer: a page of the same site fetched a median of 16 seconds earlier		92%
Within a minute after a ChatGPT user opened the page		6%

Source: the same logs (about four months of 2026). Shares of the first fetches of the site's new pages; the rest had neither. A fetch is not indexing and not a citation.

The pattern is simple. The bot reads a page, sees a link, and follows it.

1. The bot fetches a pageA page of the site it already crawls
2. It sees a link on that pageTo a page it has not fetched yet
3. It follows the linkA median of 16 seconds later; every link we saw stayed on the same site
4. First fetch of the new pageThe request carried the referer of the page it came from

Source: server logs of one older Vietnamese-language site, about four months of 2026. An observation from one site, not an experiment.

Then I tried it on my own sites

On 16 October I added plain links from the scanner pages on migurei.ai, a site ChatGPT had already cited, to two pages on pw.migurei.ai. I wrote down what would count before the links went live.

In the next 72 hours neither page got a single request from OAI-SearchBot, Claude-SearchBot, PerplexityBot or ChatGPT-User. No visit carried a referer from the scanner pages.

That looks like a refutation. It is not. I have no sign that any of those bots fetched the scanner pages in that window, and a bot cannot follow a link on a page it never reads. OAI-SearchBot also barely visits pw: in those three days it only read robots.txt. "The link does not work" and "the bot did not come" both predict zero.

What you can do

1. Link every new page from a page that crawlers already fetch: the top page, a category page, a news list. Google gives the same advice for its own crawler.
2. Keep your sitemap, but do not count on it alone to reach OpenAI.
3. Look for OAI-SearchBot in your server logs. The referer field shows which page led it where.

What this does not prove

• It is a hypothesis from one site, an observation, not an experiment.
• Every link the bot followed stayed on the same host. Whether it follows links from one site to another is untested, and my own try was too weak to tell.
• Across all first fetches in those logs, about a third had neither a referer nor a ChatGPT visit before them. I do not know where those came from.
• A fetch is not indexing, and not a citation in ChatGPT.

The full numbers are on the pw research page Is ChatGPT reading my site?