- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Removed sitemap https://www.metlife.com/investments/global/sitemap.xml
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.metlife.com/investments/global/sitemap.xml
- First snapshot of robots.txt archived
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- Added Disallow: /*&sheet= for *
- Added Disallow: /*?sheet= for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Added Disallow: /*?*&uol_app for *
- Added Disallow: /*?uol_app for *
- Removed Disallow: /*.ghtm?*&uol_app for *
- Removed Disallow: /*.ghtm?uol_app for *
- Removed Disallow: /*.htm?*&uol_app for *
- Removed Disallow: /*.htm?uol_app for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: */ab/ for *
- Added Disallow: */ab/ for Twitterbot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*/retweets for Bingbot
- Added Disallow: /*/retweets? for Bingbot
- Removed Disallow: /*/status/*/likes for Bingbot
- Removed Disallow: /*/status/*/retweets for Bingbot
- Added Disallow: /*/retweets for facebookexternalhit
- Added Disallow: /*/retweets? for facebookexternalhit
- … 6 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml