- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘*’ entirely
- Removed Allow: /wp-admin/admin-ajax.php for *
- Removed Allow: /wp-includes/css/dist/block-library/style.min.css for *
- Removed Disallow: /interactive/reading-preference for *
- Removed Disallow: /prh-internal-news/ for *
- Removed Disallow: /search/ for *
- … 5 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.lilly.com/ro/sitemap-index.xml
- Removed Disallow: */site/searchpage.jsp?st=* for *
- Removed Disallow: *?cp=*&id=*&st=* for *
- Removed Disallow: *?id*_facet*&st=* for *
- Removed Disallow: *?id*_facet*_facet*&st=* for *
- 🤖 AI Removed Disallow: *?cp=*&id=*&st=* for AmazonBot
- 🤖 AI Removed Disallow: *?id*_facet*&st=* for AmazonBot
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /account for *
- Added Disallow: /api/ for *
- Added Disallow: /auth/ for *
- Added Disallow: /library for *
- Added Disallow: /login for *
- Added Disallow: /onboarding for *
- … 8 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /become-a-customer-ppc for *
- Added sitemap https://www.reuters.com/static/video-sitemap/jp/sitemap_video_index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for CHATGPT-User
- 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-SearchBot
- 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/try-these-5-weave-tools-and-share-your-own/$ for Claude-User
- 🤖 AI Added Allow: /webinars/intro-to-weave-workflow/$ for Claude-User
- … 4 more change(s)
- Removed Disallow: /en/our-formats* for *
- Removed Disallow: /fr/nos-formats* for *
- Removed Disallow: /fr/our-formats* for *
- Formatting/comment-only change (no semantic difference)