- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2653snapshots
1482changes detected
237this week
- Added Disallow: /potash/co-op-program$ for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘Googlebot’
- Added Disallow: /*/build/ for YisouSpider
- Removed Disallow: /pl/build/ for YisouSpider
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /ar-es/shop/banner/ajax/load/ for *
- Added Disallow: /ar-es/shop/hpols_catalog/ for *
- Added Disallow: /au-en/shop/banner/ajax/load/ for *
- Added Disallow: /au-en/shop/hpols_catalog/ for *
- Added Disallow: /br-pt/shop/banner/ajax/load/ for *
- Added Disallow: /br-pt/shop/hpols_catalog/ for *
- … 32 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/blog/introducing-keyboard-accessibility-features/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/new-features-in-figma-buzz/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/5-shifts-redefining-design-systems-in-the-ai-era/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/you-never-stop-cultivating-taste/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /storage/ for *
- Removed Disallow: *utm= for *
- Removed Disallow: /api/ for *
- Removed Disallow: /backend/ for *
- Removed Disallow: /timeline for *
- Crawl-delay for *: 1 → none
- … 1 more change(s)
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_www.pinterest.com.xml
- Added sitemap https://www.reuters.com/arc/outboundfeeds/sitemap-ar-index/?outputType=xml
- Added Disallow: /j.php for *
- Added Disallow: /personal-banking/account-online/power-pay-account/ for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added rules for user-agent ‘amazon-Qbusiness’
- Added rules for user-agent ‘Applebot’
- Added rules for user-agent ‘Baiduspider’
- 🤖 AI Added rules for AI crawler Bytespider (ByteDance)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- Added rules for user-agent ‘SERankingBacklinksBot’
- … 4 more change(s)
- Added sitemap https://www.salesforce.com/ap/blog/post-sitemap.xml
- Added sitemap https://www.kraken.com/sitemaps/trading-api.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /suche/ for *
- robots.txt is no longer served (HTTP 404)
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for Claude-SearchBot
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for OAI-SearchBot
- 🤖 AI Added Allow: /es-es/blog/introducing-figma-motion/$ for PerplexityBot
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘360Spider’
- Removed all rules for user-agent ‘AhrefsBot’
- Removed all rules for user-agent ‘archive.org_bot’
- Removed all rules for user-agent ‘ArchiveBot’
- Removed all rules for user-agent ‘BingPreview’
- Removed all rules for user-agent ‘BrightEdge’
- … 26 more change(s)
- Added Allow: / for *