- Added rules for user-agent ‘meta-externalads’
- 🤖 AI Added rules for AI crawler meta-externalagent (Meta)
- 🤖 AI Added rules for AI crawler meta-externalfetcher (Meta)
- 🤖 AI Added rules for AI crawler Meta-WebIndexer (Meta)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Added sitemap https://www.lilly.com/il/sitemap-index.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://totalenergies.com/sitemap.xml
- Removed sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /assets/ for *
- Added Disallow: /content/ for *
- Added Disallow: /coveo/rest/ for *
- Added Disallow: /search-results for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /subscribe/trial-purchase/ for *
- 🤖 AI Added Allow: /de-de/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /ko-kr/blog/how-we-secure-figmas-internal-systems-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/3d-scene-generator-ai/$ for CHATGPT-User
- 🤖 AI Added Allow: /solutions/ai-brand-video-generator/$ for CHATGPT-User
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)