- Formatting/comment-only change (no semantic difference)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- 🤖 AI Added Allow: /blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
- 🤖 AI Added Allow: /de-de/blog/how-figma-stays-ahead-of-vulnerabilities-with-agents/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/code-craft-and-the-making-of-nested-folders/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/designer-framework-for-better-ai-prompts/$ for CHATGPT-User
- … 45 more change(s)
- Added sitemap https://lumalabs.ai/authors/sitemap.xml
- Added rules for user-agent ‘carbon-umbrella-bot’
- Added Allow: /story/shopping/ for ChatGPT Operator
- 🤖 AI Added Allow: /story/shopping/ for ChatGPT-User
- 🤖 AI Added Allow: /story/shopping/ for Google-Extended
- 🤖 AI Added Allow: /story/shopping/ for GPTBot
- 🤖 AI Added Allow: /story/shopping/ for OAI-SearchBot
- 🤖 AI Added Allow: /story/shopping/ for PerplexityBot
- Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
- Removed sitemap https://totalenergies.com/sitemap.xml
- Added sitemap https://www.thetradedesk.com/sitemap.xml
- Removed sitemap https://www.thetradedesk.cn/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://dotcom.ecolab.com/sitemap.xml
- Removed sitemap https://www.ecolab.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘AhrefsSiteAudit’ entirely
- 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- Blocked user-agent ‘audisto’ entirely
- Blocked user-agent ‘audisto-essential’ entirely
- Blocked user-agent ‘barkrowler’ entirely
- … 45 more change(s)
- Crawl-delay for *: none → 10
- Added sitemap http://webz.io/sitemap_index.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Formatting/comment-only change (no semantic difference)
- Added Allow: /community/gallery$ for *
- Added Allow: /community/gallery/assets$ for *
- Added Allow: /community/gallery/challenges$ for *
- Added Disallow: /community/gallery for *
- Removed Disallow: /community/gallery/* for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.bca.co.id/sitemap.xml
- Removed sitemap http://www.bca.co.id/sitemap.xml
- Formatting/comment-only change (no semantic difference)