- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5439snapshots
4304changes detected
239this week
- Removed sitemap https://arkose-labs.webflow.io/sitemap.xml
- Added Disallow: /content/dam/corp/web/unindex-documents/ for *
- Removed sitemap https://www.jlg.com/de-de/sitemap-xml
- Removed sitemap https://www.jlg.com/en-ae/sitemap-xml
- Removed sitemap https://www.jlg.com/en-au/sitemap-xml
- Removed sitemap https://www.jlg.com/en-br/sitemap-xml
- Removed sitemap https://www.jlg.com/en-gb/sitemap-xml
- Removed sitemap https://www.jlg.com/en-jp/sitemap-xml
- … 20 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Added Allow: /docs/bundle/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Removed sitemap https://www.kraken.com/sitemaps/currency-converter.xml
- Removed sitemap https://www.kraken.com/sitemaps/send-money-to-country-from-country.xml
- Removed sitemap https://www.kraken.com/sitemaps/send-money-to-country.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.bca.co.id/sitemap.xml
- Removed sitemap https://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived