- Added Disallow: /storage/ for *
- Removed Disallow: *utm= for *
- Removed Disallow: /api/ for *
- Removed Disallow: /backend/ for *
- Removed Disallow: /timeline for *
- Crawl-delay for *: 1 → none
- … 1 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_www.pinterest.com.xml
- Added sitemap https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml
- Added Disallow: /j.php for *
- Added Disallow: /personal-banking/account-online/power-pay-account/ for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added rules for user-agent ‘amazon-Qbusiness’
- Added rules for user-agent ‘Applebot’
- Added rules for user-agent ‘Baiduspider’
- 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- Added rules for user-agent ‘SERankingBacklinksBot’
- … 4 more change(s)
- Added sitemap https://www.salesforce.com/ap/blog/post-sitemap.xml
- Added sitemap https://www.kraken.com/sitemaps/trading-api.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /suche/ for *
- robots.txt is no longer served (HTTP 404)
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for Claude-SearchBot
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for OAI-SearchBot
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for PerplexityBot
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘360Spider’
- Removed all rules for user-agent ‘AhrefsBot’
- Removed all rules for user-agent ‘archive.org_bot’
- Removed all rules for user-agent ‘ArchiveBot’
- Removed all rules for user-agent ‘BingPreview’
- Removed all rules for user-agent ‘BrightEdge’
- … 26 more change(s)
- Added Allow: / for *
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.supermicro.com/sitemap_master.xml
- Added rules for user-agent ‘OAI-AdsBot’
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)