- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2643snapshots
1471changes detected
226this week
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/sightlines/$ for CHATGPT-User
- 🤖 AI Removed Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/gpt-5-6-is-now-available-in-figma-make/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/jen-dunnam-lead-design-teams/$ for Claude-SearchBot
- … 10 more change(s)
- Added Allow: /docs-static/apple-touch-icon* for *
- Added Allow: /docs-static/favicon* for *
- Added Allow: /docs-static/icon-* for *
- Added Allow: /marketing-static/apple-touch-icon* for *
- Added Allow: /marketing-static/favicon* for *
- Added Allow: /marketing-static/icon-* for *
- Added sitemap https://www.roblox.com/sitemap-ar.xml
- Added sitemap https://www.roblox.com/sitemap-de.xml
- Added sitemap https://www.roblox.com/sitemap-es.xml
- Added sitemap https://www.roblox.com/sitemap-fr.xml
- Added sitemap https://www.roblox.com/sitemap-games-ar.xml
- Added sitemap https://www.roblox.com/sitemap-games-de.xml
- … 22 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Added Disallow: /*/mp-resource/en-na/* for *
- Removed all rules for user-agent ‘Googlebot’
- Added Allow: /*/build/$ for *
- Added Allow: /build/$ for *
- Added Disallow: /*/build/ for *
- Added Disallow: /build/ for *
- Added Allow: /zh-cn/ for Baiduspider
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for CHATGPT-User
- 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/customers/botim-ai-ready-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /resource-library/claude-skills-for-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/jen-dunham-lead-design-teams/$ for Claude-SearchBot
- 🤖 AI Added Allow: /customers/botim-ai-ready-design/$ for Claude-SearchBot
- … 10 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘Googlebot-Image/1.0’ entirely
- Crawl-delay for *: 1 → none
- Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
- Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for *
- Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
- Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for DotBot
- Added Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
- Removed Disallow: /6numamre65TIZwS0ZPxJfWXMksj6o7uqs8i0hajrZzQ=.html for sistrix
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Added Disallow: /sites/default/files/e-books/pfpt-us-eb-email-security-blind-spots.pdf for *
- Added Disallow: /search?q= for Bingbot
- Removed Allow: /search?q=%23 for Bingbot
- Added Disallow: /search?q= for facebookexternalhit
- Removed Allow: /search?q=%23 for facebookexternalhit
- Added Disallow: /search?q= for Googlebot
- Removed Allow: /search?q=%23 for Googlebot
- Added Disallow: /company/blog/search for *
- Added Disallow: /docs/site-search/ for *
- Removed Disallow: /docs/search/ for *
- Added Disallow: /cartsvc/ for *
- Added Disallow: /checkout/ for *
- 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
- Added rules for user-agent ‘bingbot’
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
- … 9 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml