Who measures how Japanese sites handle AI bots, and how? A list of the public surveys (November 2026)
As far as we could confirm on 6 November 2026, four parties publish numbers on how Japanese sites handle AI bots: three outside parties, AB Tsushin (markenote), Uravation and earthdiver, and us, Parallel World (migurei.ai), making four.
This list is written by Parallel World (migurei.ai). We are in the list too. Our row is in the same format as the others, with its source.
| Who | Measured (published) | Sample | What is measured | Numbers as published | Open data | Source | Date read |
|---|---|---|---|---|---|---|---|
| AB Tsushin (markenote, abcompany Inc.) | 16 Oct 2026 | 120 major sites (19 industries) | robots.txt, llms.txt | GPTBot fully blocked: 32 of 99 sites. llms.txt: 6 of 120 sites | Yes (CSV can be downloaded) | markenote article | 6 Nov 2026 |
| Uravation (Uravation Inc.) | 23 Oct 2026 (published 31 Oct) | Official sites of the Nikkei 225 companies | robots.txt, llms.txt | llms.txt: 9 of 180 companies. robots.txt naming AI bots in its text: 9 of 184 companies | Not seen in the pages we read (totals only published) | PR TIMES release | 6 Nov 2026 |
| curumi (curumi Inc.) | (published 23 Oct 2026) | The article carries no numbers measured on Japanese sites | A how-to for writing llms.txt | None | None | curumi blog | 6 Nov 2026 |
| onetrace (AI Topica) | (published 31 Oct 2026) | Same as Uravation (Nikkei 225) | Not its own measurement: it republishes Uravation's numbers | Same as Uravation's (row above) | Same as Uravation | AI Topica article | 6 Nov 2026 |
| earthdiver (earthdiver Inc., AI MARKETING JOURNAL) | 3 Nov 2026 | Nikkei 225 (225 sites) | robots.txt, fetching with a browser-equivalent UA, llms.txt | AI search bots blocked: 0 of 98 sites with a readable robots.txt. Fetch with a browser-equivalent user agent (UA) refused: 33 of 225. llms.txt: 8 sites (denominator: see the source) | Not seen in the pages we read (totals only published) | AI MARKETING JOURNAL article | 6 Nov 2026 |
| Cloudflare Radar | Updated continuously (the chart we read covers 16-23 Oct 2026) | Global bot traffic. We have not confirmed a Japan-only figure | AI bot traffic. What we looked at is the crawl-to-refer ratio chart | Numbers not copied here (see the chart) | Yes (charts CC BY 4.0, API CC BY-NC 4.0) | Cloudflare Radar AI Insights | 23 Oct 2026 |
| Parallel World (us, migurei.ai) | Weekly (week 45: 2 Nov 2026) | Top 1,000 .jp domains by Tranco (719 with a robots.txt verdict) | robots.txt, llms.txt | GPTBot fully blocked: 76 of the 719 with a verdict. llms.txt: 30 of the 782 probed | Yes (weekly CSV) | Survey page, week 45 CSV | 2 Nov 2026 (date measured) |
We do not count curumi (no numbers on Japanese sites), onetrace (it republishes Uravation's numbers) or Cloudflare Radar (global numbers) among the four.
The outside surveys are in order of date, oldest first, then Cloudflare Radar, then us. This is not a ranking and implies no better or worse. The reasons the denominators differ are in each source; we have not checked them.
- AB Tsushin (markenote)
- Measured
- 16 Oct 2026
- Sample
- 120 major sites
- GPTBot fully blocked
- 32 of 99
- llms.txt
- 6 of 120
- Uravation
- Measured
- 23 Oct 2026
- Sample
- Nikkei 225 official sites
- llms.txt
- 9 of 180
- robots.txt naming AI bots in its text
- 9 of 184
- earthdiver
- Measured
- 3 Nov 2026
- Sample
- Nikkei 225 (225 sites)
- AI search bots blocked
- 0 of 98 readable
- Browser-equivalent UA fetch refused
- 33 of 225
- Parallel World (us)
- Measured
- 2 Nov 2026
- Sample
- Top 1,000 .jp domains by Tranco
- GPTBot fully blocked
- 76 of 719 with a verdict
- llms.txt
- 30 of 782 probed
Source: the table in the text. Samples, denominators and dates all differ.
- Who measuredIts own measurement, or another party's numbers repeated
- Which sites, whenNikkei 225, 120 major sites, top 1,000 .jp domains
- What was countedrobots.txt, llms.txt, whether a fetch worked
- What the denominator isSites that could be read, or all sites
Source: the table in the text (pages read on 6 November 2026). This is not a ranking.
What this list does not tell you
- The numbers cannot be lined up against each other. The samples (Nikkei 225, 120 major sites, the top 1,000 .jp domains by Tranco), the denominators (180, 184, 225, 99, 120, 719, 782) and the dates (16 October to 2 November 2026) all differ.
- We read each page once, on 6 November 2026 (Cloudflare Radar on 23 October). Pages may have changed since.
- Numbers from outside are copied from what each page states. We did not recompute them.
- The list is what we found. Surveys we did not find are missing.
- Most surveys that measure Japanese sites count how robots.txt and llms.txt are written.
- Surveys that measure access logs or AI answers are not included in this list.
- Cloudflare Radar measures traffic, but what we confirmed is a global chart.
About us
This article is written by the Parallel World (migurei.ai) operations team. What we do to measure sites as AI agents see them is described on the Parallel World page.
Frequently asked questions
Can the surveys of how Japanese sites handle AI bots be compared directly?
No. The samples differ (120 major sites, official sites of the Nikkei 225 companies, the top 1,000 .jp domains by Tranco) and so do the dates, from 16 October to 2 November 2026. Check the sample and the date measured, then read each survey's numbers on their own.
How many Japanese sites have an llms.txt?
It depends on the sample surveyed. Of the 120 major sites AB Tsushin (markenote) surveyed on 16 October 2026, 6 of 120; of the Nikkei 225 official sites Uravation surveyed on 23 October, 9 of 180; of the Nikkei 225 sites earthdiver surveyed on 3 November, 8 sites (denominator: see the source); of the top 1,000 .jp domains by Tranco we surveyed on 2 November, 30 of the 782 we could probe. The samples differ, so we have not turned them into percentages to line up.