- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Google-Extended (Google)
- 🤖 AI Added Allow: / for Google-Extended
- 🤖 AI Added Allow: /blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/what-the-design-to-code-loop-unlocks/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/what-matters-when-anyone-can-build/$ for CHATGPT-User
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
- … 7 more change(s)
- Added Allow: /voice$ for *
- Added Allow: /voice/$ for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Removed Disallow: /user/ for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /user/ for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘facebookexternalhit’
- Added Disallow: (empty) for *
- Added Disallow: /wp-content/uploads/wpforms/ for *
- Removed Disallow: /policies/ for *
- Added sitemap https://www.reliance.com/sitemap_index.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/loesungen/farbe-aendern/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-design-assistent/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/fonts/dm-sans/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/solutions/ai-change-color/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/solutions/ai-design-agent/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/solutions/ai-design-assistant/$ for CHATGPT-User
- … 45 more change(s)