giang's blog

Notes on how AI agents and crawlers read websites, written from the evidence we keep at Migurei.

• Cloudflare Crawler Hints automatically sends IndexNow when a response from the zone is a cache MISS (a sign that the content changed), available on every plan including Free; responses of 4xx and above are not reported. 2026-10-23 · External source
• Bing Webmaster lets site owners set the crawl rate for each hour of the day (Crawl Control; API SaveCrawlSettings, 24 numbers); Bing calls this "input" to Bingbot on crawl rate and timing, and does not promise more crawling. 2026-10-23 · External source
• According to Profound, a page cited by AI apps typically loses half of its citations 11 days after its peak (median; 78% of pages lose half within two weeks), pages cited by many apps hold on longer, and Claude Sonnet 5.5 cites newer pages (the top 10% most-cited pages have a median age of 107 days versus 164; titles containing "2026" 38% → 50%). 2026-10-23 · External source
• On the same hostname, a Worker attached to a route runs before a Worker attached to a Custom Domain; the Custom Domain Worker is treated as the origin, so a narrow route (a single path) can serve a separate file without redeploying the main Worker. 2026-10-23 · External source
• I thought my language redirect hid my Japanese pages from Google. The numbers said otherwise 2026-10-23 · Refuted
• I pasted 20 links into ChatGPT and Gemini. Their search crawlers followed none of them 2026-10-23 · Refuted
• I wrote that AI bots ignore llms.txt. Two days later I had to take it back 2026-10-23 · Refuted
• I gave Gemini my link. Sometimes it never opened it, searched the web instead, and missed 2026-10-23 · Measured in the lab
• A "Claude" came looking for my .env file. Its IP was not on Anthropic's list 2026-10-23 · Lab observation
• Our page says 60+ languages. Crawlers read 0. AI apps often say 4 2026-10-23 · Lab observation