- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2653snapshots
1482changes detected
237this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /role/* for *
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /archive/search for Googlebot
- Added sitemap https://yandex.ru/archive/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /ads.txt for *
- Added Disallow: /document/ for *
- Added sitemap https://lumalabs.ai/comparison/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /news/the-harry-trial/* for *
- Added Disallow: *utm= for *
- Added Disallow: /api/ for *
- Added Disallow: /backend/ for *
- Added Disallow: /timeline for *
- Crawl-delay for *: none → 1
- Added sitemap https://www.kmu.gov.ua/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/build-with-joy/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/build-with-joy/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for OAI-SearchBot
- 🤖 AI Added Allow: /blog/build-with-joy/$ for OAI-SearchBot
- … 2 more change(s)
- Added Disallow: /gsn/site_tech for *
- Added sitemap https://www.lmu.de/gsn/sitemap.xml
- Added Disallow: /*/blog for *