About the Parallel World scanner (migurei-ParallelWeb)

Found this crawler in your server logs?

This crawler reads only public pages, read-only, to study how AI assistants see Japanese websites. It obeys robots.txt and never submits forms or logs in. For how your own site looks to AI, see How an AI assistant learns about your company.

Overview

Parallel World is a service run by Migurei (HBLAB). It measures how usable a website is for AI agents, from training crawlers to agents that complete transactions. This scanner fetches the public pages that measurement needs. For background on Parallel World, see also the article "How an AI assistant learns about your company".

User-Agent

The scanner identifies itself with this User-Agent (except as described under "Exception" below):

migurei-ParallelWeb/0.1 (+https://migurei.ai/ja/scanner; research@migurei.ai)

What the scanner accesses

Pages

The top page, at most one category page, at most one detail page, plus robots.txt, the sitemap, llms.txt and files under .well-known

Method

Pages are fetched two ways: as raw HTML, and rendered in a headless browser that runs JavaScript.

Loads

Each page is loaded 3 times, to account for lazy loading and A/B tests.

Rendering

Language ja-JP, time zone Asia/Tokyo, fixed viewport

Frequency

Once a week

Spacing

Requests to the same domain are sent at least 2–3 seconds apart.

Volume

The scanner itself sends no more than 50 requests per scan. Resources that a page loads on its own while rendering, such as images and scripts, are counted against a separate limit.

robots.txt

The scanner obeys robots.txt for the token migurei-ParallelWeb. If you do not want it to access your site at all, add the following to robots.txt:

User-agent: migurei-ParallelWeb
Disallow: /

robots.txt is the first thing the scanner fetches. If access to the top page (/) is disallowed, the scan ends there, and the sitemap, llms.txt and .well-known files are not fetched. In that case the User-Agent reachability check and the external checks below do not run either. The only thing recorded is that robots.txt disallows access.

What the scanner does not do

• Submit forms
• Add to cart or place orders
• Sign up or log in
• Solve CAPTCHAs or evade bot detection
• Access pages disallowed in robots.txt
• Change its User-Agent or IP address to get around a block (the reachability check under "Exception" below only checks whether AI services can reach the site; it does not get around blocks)

If access is blocked, the scanner records only that it was blocked.

Exception: reachability check with AI services' User-Agents

To check whether AI services can reach your site, the scan sends the following requests:

• Top page only. One request each with the User-Agents of major AI services (6–8 of them).
• User-Agent strings are the ones each company publishes. For some User-Agents whose full string is not published (for example, Anthropic's Claude-User), we use the string seen in the logs of our own test environment.
• Only the HTTP status code, whether a bot challenge page (such as a CAPTCHA) appears and of what type, and time to first byte (TTFB) are recorded. Page content is not stored.
• No retries and no IP changes. Requests come from the same IP address as the scanner, so only User-Agent-based blocking can be detected.
• These requests do not carry migurei-ParallelWeb in their User-Agent, so in your server logs they look like visits from AI services. They are not visits by the AI services themselves. This scanner sent them as part of the same scan.

External checking services

Cloudflare URL Scanner results are published on Cloudflare Radar. If you do not want this, the robots.txt rule above also stops the external checks.

As part of a scan, we ask the following external services to check the site. Those requests come from each service, with that service's own User-Agent.

• Cloudflare URL Scanner (at most once per 7 days per site)
• Google PageSpeed Insights

AI bots this scanner checks (as of registry bots-2026-10-19)

This is the list the scanner uses to tell whether a request in your server logs comes from a given company's bot. It is generated automatically from the Parallel World bot registry.

Verification: "Published IP list" means matched against the IP address ranges the operator publishes; "User-Agent only" means judged from the User-Agent string alone.

* The operator's official documentation does not confirm this User-Agent string, or the name is a robots.txt token only (it does not appear in logs as a request source).

A "—" in the official documentation column means we could not find an official explanatory page from the operator.

Contact

For questions about this scanner, contact:

research@migurei.ai

Research notes

We publish research recording when AI bots visited our test site (pw.migurei.ai). Bots are identified from server logs and the same verified IP ranges as this scanner's bot list. It contains no scan results from customer sites.

• When do AI search bots visit a new subdomain? Records from pw.migurei.ai