- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: */Order-MyOrderDetails for *
- Added Disallow: */Wishlist-ShowAll for *
- Added Disallow: */addressbook for *
- Added Disallow: */mywardrobe for *
- Added Disallow: */orderconfirm for *
- Added Disallow: */orderdetails for *
- … 7 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /Restaurants-g55711-Dallas_Texas.html for Google-Extended
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: *UCPthyssenkruppAG* for *
- Added Disallow: /_api for *
- Added Disallow: /_media/ for *
- Added Disallow: /assets.files/ for *
- Added sitemap https://www.thyssenkrupp.com/sitemap.xml
- 🤖 AI Added Allow: /blog/how-we-built-generative-plugins-and-shaders/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no2/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/sightlines-issue-no2/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/sightlines-issue-no2/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/blog/sightlines-issue-no2/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/blog/sightlines-issue-no2/$ for CHATGPT-User
- … 39 more change(s)
- Added rules for user-agent ‘APSMeridianBot’
- Removed all rules for user-agent ‘*’
- Removed sitemap https://www.copart-sitemaps.com/ar/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/es/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/fr-CA/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/pl/sitemap-index.xml
- Removed sitemap https://www.copart-sitemaps.com/ru/sitemap-index.xml
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /nubank-parque for *
- Added Disallow: /nubank-parque for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AIWebIndex’ entirely
- Blocked user-agent ‘AIWebIndex-Agent’ entirely
- Blocked user-agent ‘CragCrawler’ entirely
- Blocked user-agent ‘Crawlspace’ entirely
- Blocked user-agent ‘Datenbank Crawler’ entirely
- Blocked user-agent ‘ExaSearchBot’ entirely
- … 9 more change(s)
- Added Disallow: (empty) for *
- Removed Disallow: *UCPthyssenkruppAG* for *
- Removed Disallow: /_api for *
- Removed Disallow: /_media/ for *
- Removed Disallow: /assets.files/ for *
- Removed Disallow: /de/home-2604 for *
- … 4 more change(s)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)