- First snapshot of robots.txt archived
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2673snapshots
1503changes detected
231this week
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- First snapshot of robots.txt archived
- Added Disallow: /gtwy/api/ecom/purchase-history/order-details/* for *
- Added Disallow: /gtwy/api/ecom/purchase-history/order-list* for *
- Added Disallow: /tsc/cart* for *
- Added Disallow: /tsc/checkout* for *
- Added Disallow: /tsc/orderconfirmation* for *
- Added Disallow: /tsc/orders* for *
- Added Disallow: /*?*&searchQuery=* for *
- Added Disallow: /*?searchQuery=* for *
- Added sitemap https://www.sap.com/sitemap-us-int-only-siteimprove.xml
- Added sitemap https://www.sap.com/sitemap_countries_siteimprove.xml
- Added sitemap https://www.sap.com/sitemap_us_int_only_siteimprove.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://arkose-labs.webflow.io/sitemap.xml
- Added Disallow: /content/dam/corp/web/unindex-documents/ for *
- Removed sitemap https://www.jlg.com/de-de/sitemap-xml
- Removed sitemap https://www.jlg.com/en-ae/sitemap-xml
- Removed sitemap https://www.jlg.com/en-au/sitemap-xml
- Removed sitemap https://www.jlg.com/en-br/sitemap-xml
- Removed sitemap https://www.jlg.com/en-gb/sitemap-xml
- Removed sitemap https://www.jlg.com/en-jp/sitemap-xml
- … 20 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added Allow: /docs/bundle/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://www.kraken.com/sitemaps/currency-converter.xml
- Removed sitemap https://www.kraken.com/sitemaps/send-money-to-country-from-country.xml
- Removed sitemap https://www.kraken.com/sitemaps/send-money-to-country.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml