- Added Disallow: */lp/ for *
- Added Disallow: */sac/*/*/*/appref=* for *
- Added Disallow: */shop/*/scr/* for *
- Added Disallow: */shop/dell-parts/ar/*/platform=* for *
- Added Disallow: */shop/dell-parts/partsbytype/* for *
- Added Disallow: */shop/partsbytype/* for *
- … 6 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-SearchBot
- 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-User
- 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for OAI-SearchBot
- 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for PerplexityBot
- Removed Disallow: /en/medialibrary* for *
- Removed Disallow: /fr/mediatheque* for *
- Removed Disallow: /fr/recherche* for *
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_catholic_christmas_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_catholic_protestant_christmas_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_christmas_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261225_family_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_day_after_christmas_day_www.pinterest.com.xml
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261226_national_day_of_thanksgiving_www.pinterest.com.xml
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: */ar/*/*?appliedRefinements=* for *
- Removed Disallow: */campaign/lp/ for *
- Removed Disallow: */learn/lp/ for *
- Removed Disallow: */legal/lp/ for *
- Removed Disallow: */lp/ for *
- Removed Disallow: */shop/*/*/apd/*/*/- for *
- … 2 more change(s)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: */shopping/ for *
- Removed Disallow: */sac/*/*/*/appref=* for *
- Removed Disallow: */shop/dell-parts/ar/*/platform=* for *
- Removed Disallow: */shop/dell-parts/partsbytype/* for *
- Removed Disallow: */shop/partsbytype/* for *
- Removed Disallow: */shop/pfydresults/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)