- Added Allow: /en-gb/blog/ for *
- Removed Disallow: /support/*/ServiceTag/* for *
- Removed Disallow: /support/*/servicetag/* for *
- Removed Disallow: /support/Search/ for *
- Removed Disallow: /support/article{lwp}/* for *
- Removed Disallow: /support/incidents-online/*/contactus/product/ for *
- … 2 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Added Disallow: /c/ for Googlebot
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Added Disallow: /members/*/mem for *
- Added Disallow: /providers/*/duals for *
- Added Disallow: /providers/*/medicaid for *
- Added Disallow: /providers/*/medicare for *
- Removed Disallow: (empty) for *
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /assets/document/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: (empty) for *
- Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Removed Disallow: /members/*/mem for *
- Removed Disallow: /providers/*/duals for *
- Removed Disallow: /providers/*/medicaid for *
- Removed Disallow: /providers/*/medicare for *
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /nu-empresas-sebrae-conteudos for *
- Added Disallow: /nu-empresas-sebrae-conteudos for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /healthz/available for *
- Added Disallow: /graduatecenter/site_tech for *
- Added sitemap https://www.lmu.de/graduatecenter/sitemap.xml
- 🤖 AI Added Allow: /*Drinking-Water for OAI-SearchBot
- 🤖 AI Added Allow: /*Drinking-water for OAI-SearchBot
- 🤖 AI Added Allow: /*drinking-Water for OAI-SearchBot
- 🤖 AI Added Allow: /*drinking-water for OAI-SearchBot
- Added Disallow: /onboarding/* for *
- Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Added Disallow: /members/*/mem for *
- Added Disallow: /providers/*/duals for *
- Added Disallow: /providers/*/medicaid for *
- Added Disallow: /providers/*/medicare for *
- Removed Disallow: (empty) for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for CHATGPT-User
- 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for CHATGPT-User
- 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for Claude-SearchBot
- 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for Claude-SearchBot
- 🤖 AI Added Allow: /legal/figma-agent-credit-promo/$ for Claude-User
- 🤖 AI Removed Allow: /legal/agent-credit-promo-project/$ for Claude-User
- … 4 more change(s)
- Added sitemap https://www.meta.com/sitemap/www_meta_com_blog_news_sitemap.xml.gz
- Added sitemap https://weather.com/api/payload/v1/sitemaps/en-US/news.xml
- Added sitemap https://weather.com/en-US/sitemaps/bios-sitemap.xml
- Added sitemap https://weather.com/en-US/sitemaps/collections-manual.xml
- Added sitemap https://weather.com/en-US/sitemaps/news-0-1300.xml
- Removed sitemap https://weather.com/en-CA/sitemaps/sitemap.xml
- Removed sitemap https://weather.com/en-GB/sitemaps/sitemap.xml
- … 5 more change(s)
- Added rules for user-agent ‘facebookexternalhit’
- Added rules for user-agent ‘LinkedInBot’
- Added rules for user-agent ‘WhatsApp’
- Formatting/comment-only change (no semantic difference)