- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked anthropic-ai (Anthropic) entirely
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- Blocked user-agent ‘atlassian-bot’ entirely
- Blocked user-agent ‘AwarioRssBot’ entirely
- Blocked user-agent ‘AwarioSmartBot’ entirely
- … 29 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- 🤖 AI Blocked meta-externalfetcher (Meta) entirely
- 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
- Blocked user-agent ‘meta-webindexer/1.1’ entirely
- Removed Disallow: /*/scripts/ for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: (empty) for *
- Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
- Removed Disallow: /marketplace/az/* for *
- Removed Disallow: /marketplace/ca/* for *
- Removed Disallow: /marketplace/fl/* for *
- Removed Disallow: /marketplace/ga/* for *
- … 24 more change(s)
- Removed Disallow: /collaborating-with-slack for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /who-we-are/our-stories/why-rheumatoid-arthritis-patients-need-more-options.html for *
- Added Disallow: /who-we-are/our-stories/why-rheumatoid-arthritis-patients-need-more-options.html for Googlebot-Video
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Removed Allow: /de-de/vorlagen/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/vorlagen/affinitatsdiagramm-beispiele/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/vorlagen/agile-roadmap-vorlage/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/vorlagen/angebots-und-nachfragekurve/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/vorlagen/beispiel-kanban-board/$ for CHATGPT-User
- 🤖 AI Removed Allow: /de-de/vorlagen/betriebsablaufe/$ for CHATGPT-User
- … 45 more change(s)
- Added Disallow: /sites/default/files/e-books/pfpt-us-eb-buyers-guide-choosing-email-collaboration-security-ai-era.pdf for *
- Removed sitemap https://www.pinterest.com/v3_sitemaps/christmas_new_year_eve_pin_www.pinterest.com.xml
- Added Disallow: /media/sites/js/bundles for *
- Added Allow: /unauth-mweb/assets/ for *
- Removed Allow: /football-sitemap.xml for *
- Removed Allow: /images-sitemap.xml for *
- Removed sitemap https://chatgpt.com/football-sitemap.xml
- Removed sitemap https://chatgpt.com/images-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://tidal.com/sitemaps/sitemap-index-playlists.xml
- Added sitemap https://tidal.com/sitemaps/sitemap-index-tracks.xml
- Added sitemap https://tidal.com/sitemaps/sitemap-index-videos.xml
- robots.txt is no longer served (HTTP 404)
- Added Allow: /realestateandhomes-search/*price-na- for *
- Added Disallow: /realestateandhomes-search/*%2C for *
- Added Disallow: /realestateandhomes-search/*%2c for *
- Added Disallow: /realestateandhomes-search/*, for *
- Added Disallow: /realestateandhomes-search/*/price- for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*search%5Bkeywords%5D= for *
- Added Disallow: /*search[keywords]= for *
- Added Disallow: /api/ for *
- Added Disallow: /en/rechercher for *
- Added Disallow: /graphql for *
- Added Disallow: /rechercher for *
- Formatting/comment-only change (no semantic difference)