- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /industries/federal/fedcat/ibm-automation-services-for-government for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for Claude-User
- 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for OAI-SearchBot
- 🤖 AI Added Allow: /blog/what-it-takes-to-build-great-products-now/$ for PerplexityBot
- Added sitemap https://www.flickr.com/sitemap/cameras/sitemap-cameras-00000000.xml.gz
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261031_halloween_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261122_sunday_of_the_dead_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261123_day_off_for_alphabet_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261124_guru_nanak_jayanti_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261124_guru_nanak_s_birthday_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261124_vietnam_culture_day_www.pinterest.com.xml
- … 17 more change(s)
- Blocked user-agent ‘Shap-User’ entirely
- Blocked user-agent ‘ShapBot’ entirely
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /FlexibleOnlineQuote/ for *
- Removed Disallow: /OnlineQuote/ for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.orange.com/en/sitemap.xml
- Added sitemap https://www.orange.com/fr/sitemap.xml
- Added Disallow: /what-we-do/our-strategy/sustainability/advocacy-and-political-activity/advocacy-releases.html for *
- Added Disallow: /what-we-do/our-strategy/sustainability/advocacy-and-political-activity/advocacy-releases/ for *
- Removed Disallow: /sustainability/advocacy-and-political-activity/advocacy-releases.html for *
- Removed Disallow: /sustainability/advocacy-and-political-activity/advocacy-releases/ for *
- Added Disallow: /*/UsaaPartner-CreateUSAACookie for *
- Added Disallow: /*/account for *
- Added Disallow: /*/actuator for *
- Added Disallow: /*/callback for *
- Added Disallow: /*/cart for *
- Added Disallow: /*/checkout for *
- … 26 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.capgemini.com/au-en/sitemap_index.xml
- Added sitemap https://www.capgemini.com/ch-en/sitemap_index.xml
- Added sitemap https://www.capgemini.com/co-es/sitemap_index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /print/ for *
- Removed Disallow: */sac/*/*/*/appref=* for *
- Removed Disallow: */shop/*/apd/* for *
- Removed Disallow: */shop/*/ar/* for *
- Removed Disallow: */shop/*/ar/*/appref=* for *
- Added sitemap https://www.aliexpress.com/sitemap/en-https-item-item-2026-P0-sitemap-index.xml
- Added sitemap https://www.aliexpress.com/sitemap/en-https-item-item-2026-P1-sitemap-index.xml
- Added sitemap https://www.aliexpress.com/sitemap/en-https-mlh-2026-sitemap-index.xml
- Added sitemap https://www.aliexpress.com/sitemap/en-https-spot-2026-sitemap-index.xml
- Added sitemap https://www.aliexpress.com/sitemap/en-https-wholesale-2025-p-transactional-local-sitemap-index.xml
- Added sitemap https://www.aliexpress.com/sitemap/en-https-wholesale-2025-w-brand-local-sitemap-index.xml
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://support.southwest.com/helpcenter/sfsites/c/resource/HelpCenter_Sitemap