- Added Disallow: *UCPthyssenkruppAG* for *
- Added Disallow: /_api for *
- Added Disallow: /_media/ for *
- Added Disallow: /assets.files/ for *
- Added Disallow: /de/home-2604 for *
- Added Disallow: /de/newsroom/pressemeldungen/press-release- for *
- … 4 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
- Added sitemap https://www.dailymail.com/video-new-sitemap.xml
- Removed sitemap https://www.dailymail.com/sitemap-authors.xml
- Removed sitemap https://www.dailymail.com/sitemap-channels.xml
- Removed sitemap https://www.dailymail.com/sitemap-columnist.xml
- Removed sitemap https://www.dailymail.com/sitemap-facts-year~2021.xml
- Removed sitemap https://www.dailymail.com/sitemap-facts-year~2022.xml
- … 5 more change(s)
- Blocked user-agent ‘AwarioRssBot’ entirely
- Blocked user-agent ‘AwarioSmartBot’ entirely
- Blocked user-agent ‘BacklinkBot’ entirely
- Blocked user-agent ‘Brandwatch’ entirely
- Blocked user-agent ‘CognitiveSEO’ entirely
- Blocked user-agent ‘DataForSeoBot’ entirely
- … 13 more change(s)
- 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /uk-tax-strategy/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-SearchBot
- 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-SearchBot
- 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-User
- 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-User
- … 4 more change(s)
- Added Allow: /strategist for A6-Indexer
- Added Allow: /strategist for AhrefsBot
- 🤖 AI Added Allow: /strategist for AI2Bot
- Added Allow: /strategist for AI2Bot-DeepResearchEval
- Added Allow: /strategist for Ai2Bot-Dolma
- Added Allow: /strategist for aiHitBot
- … 45 more change(s)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Removed sitemap https://www.pinterest.com/v3_sitemaps/ideas_hub_trending_ideas_terms_sitemap_www.pinterest.com.xml
- Added Disallow: /motorist/oils-lubricants/rimula-truck-heavy-duty-engine-oil/shell-starship/immersion-cooling-battery-technology.html for *
- Added Disallow: /motorist/oils-lubricants/rimula-truck-heavy-duty-engine-oil/shell-starship/immersion-cooling-battery-technology/ for *
- Added Allow: /analytics/keywordoverview/* for *
- Removed Allow: /analytics/keywordoverview/*?q=* for *
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: (empty) for *
- Removed Disallow: *UCPthyssenkruppAG* for *
- Removed Disallow: /_api for *
- Removed Disallow: /_media/ for *
- Removed Disallow: /assets.files/ for *
- Removed Disallow: /de/home-2604 for *
- … 4 more change(s)
- 🤖 AI Added Allow: /de-de/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-es/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /ja-jp/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /ko-kr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /pt-br/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
- … 45 more change(s)
- Added sitemap https://www.ucl.ac.uk/about/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/alumni/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/arts-humanities/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/bartlett/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/brain-sciences/sitemap.xml
- Added sitemap https://www.ucl.ac.uk/brand-and-experience/sitemap.xml
- … 45 more change(s)
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /api/article-search for *
- Added Disallow: /xid for *
- Removed sitemap https://video.foxnews.com/videositemap.xml
- Blocked user-agent ‘AhrefsBot’ entirely
- Blocked user-agent ‘BLEXBot’ entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked Claude-SearchBot (Anthropic) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- … 15 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://lumalabs.ai/case-studies/sitemap.xml
- Added Allow: /?* for *
- Removed Disallow: /2012-privacy-policy for *
- Added sitemap https://duckduckgo.com/duckduckgo-help-pages/sitemap.xml
- Added Allow: /?display=wide for *
- Added Allow: /wp-admin/admin-ajax.php for *
- Added Allow: /wp-content/uploads/ for *
- Added Disallow: /*/embed/ for *
- Added Disallow: /*?*p=* for *
- Added Disallow: /*?p= for *
- … 12 more change(s)