- Added sitemap https://lumalabs.ai/comparison/sitemap.xml
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- robots.txt is no longer served (HTTP 404)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /news/the-harry-trial/* for *
- Added Disallow: *utm= for *
- Added Disallow: /api/ for *
- Added Disallow: /backend/ for *
- Added Disallow: /timeline for *
- Crawl-delay for *: none → 1
- Added sitemap https://www.kmu.gov.ua/sitemap.xml
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/build-with-joy/$ for CHATGPT-User
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/build-with-joy/$ for Claude-SearchBot
- 🤖 AI Added Allow: /blog/7-questions-from-lc/$ for OAI-SearchBot
- 🤖 AI Added Allow: /blog/build-with-joy/$ for OAI-SearchBot
- … 2 more change(s)
- Added Disallow: /gsn/site_tech for *
- Added sitemap https://www.lmu.de/gsn/sitemap.xml
- Added Disallow: /*/blog for *
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Added Disallow: /potash/co-op-program$ for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘Googlebot’
- Added Disallow: /*/build/ for YisouSpider
- Removed Disallow: /pl/build/ for YisouSpider
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /ar-es/shop/banner/ajax/load/ for *
- Added Disallow: /ar-es/shop/hpols_catalog/ for *
- Added Disallow: /au-en/shop/banner/ajax/load/ for *
- Added Disallow: /au-en/shop/hpols_catalog/ for *
- Added Disallow: /br-pt/shop/banner/ajax/load/ for *
- Added Disallow: /br-pt/shop/hpols_catalog/ for *
- … 32 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /de-de/blog/introducing-keyboard-accessibility-features/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/new-features-in-figma-buzz/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/5-shifts-redefining-design-systems-in-the-ai-era/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/you-never-stop-cultivating-taste/$ for CHATGPT-User
- … 45 more change(s)