- 🤖 AI Added Allow: /de-de/loesungen/anzeigen-ersteller/facebook/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-app-icon-generator/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-dateireiniger/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/loesungen/ki-schaltflaechengenerator/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/customers/how-adyen-is-rebuilding-design/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/soluciones/creador-de-anuncios/facebook/$ for CHATGPT-User
- … 45 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- Added sitemap https://blog.suno.com/sitemap.xml
- Added sitemap https://suno.com/music-library-sitemap/sitemap.xml
- Added sitemap https://suno.com/products-sitemap/sitemap.xml
- Added sitemap https://suno.com/tools-sitemap/sitemap.xml
- Removed sitemap https://suno.com/blog-sitemap.xml
- Removed sitemap https://suno.com/music-library-sitemap.xml
- … 2 more change(s)
- Added sitemap https://www.meta.com/sitemap/www_meta_com_blog_news_sitemap.xml.gz
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.arrow.com/commerce/sitemap/categories/sitemap-categories-index.xml
- Added sitemap https://www.arrow.com/commerce/sitemap/manufacturers/sitemap-manufacturers-index.xml
- Added sitemap https://www.arrow.com/commerce/sitemap/products/sitemap-products-index.xml
- Added sitemap https://www.arrow.com/commerce/sitemap/referencedesigns/sitemap-referencedesign-index.xml
- Added sitemap https://www.arrow.com/sitemap.xml
- Removed sitemap https://www.arrow.com/ais/msembedded/sitemap_index.xml
- … 28 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /blue-lane-freight/blf-request for *
- Removed sitemap https://www.fastenal.com/static-assets/sitemaps/fcom-products-16.xml
- Removed sitemap https://www.fastenal.com/static-assets/sitemaps/fcom-products-17.xml
- Removed sitemap https://www.fastenal.com/static-assets/sitemaps/fcom-products-18.xml
- Removed sitemap https://www.fastenal.com/static-assets/sitemaps/fcom-products-19.xml
- Removed sitemap https://www.fastenal.com/static-assets/sitemaps/fcom-products-20.xml
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
- Blocked user-agent ‘Y!J-ASR’ entirely
- Blocked user-agent ‘YandexBot’ entirely
- Formatting/comment-only change (no semantic difference)
- Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for "008"
- Added Allow: /articles/z-fold8-vs-previous-samsung-foldables-how-battery-performance-efficiency-and-display-experience-have-evolved for "008"
- Added Allow: /sponsored/ for *
- Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for AddSearchBot
- Added Allow: /articles/z-fold8-vs-previous-samsung-foldables-how-battery-performance-efficiency-and-display-experience-have-evolved for AddSearchBot
- Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for AgentDataBot
- … 45 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /*/form/* for *
- Removed Disallow: /*/locations$ for *
- Removed Disallow: /*/webform/* for *
- Removed Disallow: /*?autotranslate=* for *
- Removed Disallow: /contact/ for *
- Removed Disallow: /de-de/investoren/document-download-tool? for *
- … 22 more change(s)
- Added Disallow: */page-template-preview/ for *
- Added Disallow: */page-template-preview/ for facebookexternalhit
- Added Disallow: */page-template-preview/ for LinkedInBot
- Added Disallow: */page-template-preview/ for Twitterbot
- Added Disallow: */page-template-preview/ for WhatsApp
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261207_the_feast_of_st_ambrose_milan_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261224_yap_constitution_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_2nd_christmas_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_catholic_protestant_boxing_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_day_of_goodwill_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_synaxis_of_the_mother_of_god_www.pinterest.com.xml
- … 1 more change(s)