- 🤖 AI Added Allow: /$ for ChatGPT-User
- 🤖 AI Added Allow: /africa$ for ChatGPT-User
- 🤖 AI Added Allow: /africa/$ for ChatGPT-User
- 🤖 AI Added Allow: /asia$ for ChatGPT-User
- 🤖 AI Added Allow: /asia/$ for ChatGPT-User
- 🤖 AI Added Allow: /canada$ for ChatGPT-User
- … 45 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
- 🤖 AI Removed Disallow: /*?* for Applebot-Extended
- 🤖 AI Removed Disallow: /*?* for ChatGPT-User
- 🤖 AI Removed Disallow: /*?* for Claude-SearchBot
- 🤖 AI Removed Disallow: /*?* for Claude-User
- 🤖 AI Removed Disallow: /*?* for ClaudeBot
- 🤖 AI Removed Disallow: /*?* for Cohere-ai
- … 4 more change(s)
- Added sitemap https://www.autodesk.com/href-sitemap-fusion-en-us.xml
- Removed sitemap https://www.autodesk.com/content/autodesk/global/en.sitemap.solutions-sitemap.xml
- 🤖 AI Blocked Applebot-Extended (Apple) entirely
- 🤖 AI Blocked Bytespider (ByteDance) entirely
- 🤖 AI Blocked CCBot (Common Crawl) entirely
- 🤖 AI Blocked ClaudeBot (Anthropic) entirely
- 🤖 AI Blocked Google-Extended (Google) entirely
- 🤖 AI Blocked GPTBot (OpenAI) entirely
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.flipkart.com/sitemap_o_clp-browse.xml.gz
- Added sitemap https://www.flipkart.com/sitemap_t_tag-browse.xml.gz
- 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
- Added Disallow: /finance/banks/*/reviews*? for *
- Added Disallow: /support2/sport for *
- Removed Disallow: /finance/banks/*/reviews*?* for *
- Removed Disallow: /support2/sport* for *
- 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
- 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
- Added Disallow: /id/Forms/Ext/event-bca for *
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*?app_name= for *
- Formatting/comment-only change (no semantic difference)
- Removed Disallow: /api/article-search for *
- Removed Disallow: /xid for *
- Added sitemap https://video.foxnews.com/videositemap.xml
- Removed sitemap https://www.instagram.com/sitemap/ig_seo_location_sitemap.xml.gz
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-au-votc-2026.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-br-votc-2026.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-de-votc-2026.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-es-votc-2026.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-fr-votc-2026.pdf for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-it-votc-2026.pdf for *
- … 2 more change(s)
- 🤖 AI Unblocked meta-externalagent (Meta) - rules removed
- 🤖 AI Unblocked Meta-WebIndexer (Meta) - rules removed
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap https://www.stonex.com/en-gb/sitemap-b2b.xml
- Added sitemap https://www.stonex.com/en-gb/sitemap-b2c.xml
- Added sitemap https://www.stonex.com/en-us/sitemap-b2b.xml
- Added sitemap https://www.stonex.com/en/sitemap-b2b.xml
- Added sitemap https://www.stonex.com/pt-br/sitemap-b2b.xml
- Removed sitemap https://www.stonex.com/en/sitemap.xml
- … 1 more change(s)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/blog/source-material-zach-lieberman/$ for CHATGPT-User
- 🤖 AI Added Allow: /fr-fr/customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
- 🤖 AI Added Allow: /ko-kr/blog/source-material-zach-lieberman/$ for CHATGPT-User
- 🤖 AI Added Allow: /pt-br/blog/source-material-zach-lieberman/$ for CHATGPT-User
- 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for Claude-SearchBot
- … 19 more change(s)
- Removed Disallow: /sites/default/files/asset-library/documents/library/documents/annual_reports/2000-management-report-brands-de.pdf for *
- Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_halloween_www.pinterest.com.xml
- Formatting/comment-only change (no semantic difference)
- Added Allow: */myaccount/signin for *
- Added Allow: */myaccount/signin for bingbot