How Japan's top 1,000 sites treat AI bots: the weekly numbers (November 2026)
On Monday 2 November 2026 I counted the same 1,000 top .jp domains for the second week running, and three numbers (76 sites blocking GPTBot, 30 with a real llms.txt, 63 answering 403 to our robots.txt request) each moved by one or two sites from the week before. The interesting part is not the movement but where the numbers live.
A second Monday, almost the same answer
Between 04:17 and 06:11 Japan time on 2 November 2026, my scanner fetched robots.txt and /llms.txt from the same 1,000 domains as the Monday before: the first 1,000 .jp domains, in rank order, of Tranco list W3ZV9. I expected something to move. Almost nothing did.
- 1,000 domainsThe same Tranco list every week (W3ZV9)
- One GET of /robots.txt eachSigned requests from a Japanese IP; read-only
- 719 give a per-bot verdict403, 5xx and a robots.txt on another domain are left out
- Count who disallows "/"GPTBot: 76 of 719 on 2 November, 74 on 26 October
Source: the weekly public CSV of the census, top 1,000 .jp domains, 26 October and 2 November 2026. The denominators differ between weeks and between rows.
Three numbers, two Mondays
- GPTBot. On 2 November 2026, 76 of the 719 sites whose robots.txt gave a verdict (10.6%) disallow GPTBot from the top page (/). On 26 October 2026: 74 of 719 (10.3%).
- llms.txt. On 2 November, 30 of the 782 sites we could probe on their own domain (3.8%) have a real /llms.txt file. On 26 October: 29 of 777 (3.7%).
- 403 on robots.txt. On 2 November, 63 of the 792 sites that answered from their own domain (8.0%) returned 403 to our signed request for robots.txt. On 26 October: 61 of 788 (7.7%).
The series, week 44 to week 45: GPTBot 74 → 76, llms.txt 29 → 30, robots.txt 403 61 → 63.
- GPTBot blocked in robots.txt
- Week 44 (26 Oct)
- 74 of 719
- Week 45 (2 Nov)
- 76 of 719
- A real llms.txt (own domain)
- Week 44 (26 Oct)
- 29 of 777
- Week 45 (2 Nov)
- 30 of 782
- robots.txt answered 403 to our scanner
- Week 44 (26 Oct)
- 61 of 788
- Week 45 (2 Nov)
- 63 of 792
Source: the weekly public CSV of the census, top 1,000 .jp domains, 26 October and 2 November 2026. Two weeks are two points, not a trend.
The full table, with the other 39 bots, is on the research page: https://pw.migurei.ai/en/research/japanese-sites-and-ai-bots. This week's data as a file: https://pw.migurei.ai/data/japanese-sites-and-ai-bots/2026-W45.csv.
719 is 719, but not the same 719
The GPTBot denominator is 719 in both weeks. I assumed that meant the same sites. It does not. A site that fails to answer on a given Monday drops out of the group with a verdict, and another one comes back in. In my recount, two sites really changed their own GPTBot line between the two Mondays, and two others entered or left the 719 through loading errors and cancelled each other out. So 74 → 76 is a small real change inside a group that is slightly different each week.
The same numbers, three homes
These numbers are published in more than one place, and a search engine does not treat the places alike. On 31 October 2026, 72 hours after I submitted the sitemaps, Google had 17 of the 28 URLs of this blog in its index and Bing 20 of 28. For the same kind of submission to a separate project host on GitHub Pages, Google answered "URL is unknown to Google" for all 11 URLs. That is why the research page on pw is the original and this post only tells the story.
What you can do
- Open your own /robots.txt and /llms.txt in a plain browser. A page that answers 200 but is not the file is not an llms.txt; the CSV counts those separately.
- If your robots.txt answers 403 to automated requests, a bot cannot read your rules at all. On object storage a 403 can also just mean the file is missing.
- When you compare your site with these figures, compare the denominator first: 719, 782 and 792 are three different groups of sites.
What these numbers do not say
They say what 719 robots.txt files ask of GPTBot for the top page, one week apart; they do not say that GPTBot obeys, that it visits, or that this is how Japan's web behaves.
Sources
This is an observation from our own lab; the raw data is not published here.
The measurements behind this are on Parallel World
What this does not prove
Measured and aggregated by us, with no independent check. This week's counts came from two scanner runs (the first stopped part-way); an independent recount of the GPTBot and robots.txt 403 figures gave the same result.
Two weeks are two points, not a trend. A move of one or two sites is within what loading errors alone can produce.
The 1,000 domains are the top of one ranking (Tranco list W3ZV9), not Japan's web. 719 of them gave a robots.txt verdict; the other 281 are not in the denominator of the GPTBot figure.
robots.txt is a request, not a block, and the check covers the top page (/) only. It does not show what GPTBot does.
The 403 is how a site treated our own signed scanner; it is not how it treats GPTBot, and on object storage it can mean a missing file.
The two hosts differ in age, Search Console property, internal links and resubmission, so the index counts do not show that the host alone made the difference. Being in an index is not the same as an AI app naming the page.