NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Check Point checkpoint.com modified +31 −0
  • Added rules for user-agent ‘AndiBot’
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • Added rules for user-agent ‘ExaBot’
  • … 7 more change(s)
view diff → snapshot
Allstate allstate.com modified +17 −4
  • Added Disallow: /allstate-benefits/* for *
  • Added Disallow: /ddt-test-pages* for *
  • Added Disallow: /es/_test* for *
  • Added Disallow: /es/allstate-benefits/* for *
  • Added Disallow: /es/auto-insurance/cars-trucks for *
  • Added Disallow: /es/credit for *
  • … 13 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /de-de/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/the-power-of-product-based-learning/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
ChatGPT chatgpt.com modified +5 −0
  • Added Allow: /*/dots for *
  • Added Allow: /*/sites for *
  • Added Allow: /*/space for *
  • Added Allow: /dots for *
  • Added Allow: /space for *
view diff → snapshot
TotalEnergies totalenergies.com modified +1 −1
  • Added sitemap https://prd-backoffice.totalenergies.com/sitemap.xml
  • Removed sitemap https://totalenergies.com/sitemap.xml
view diff → snapshot
GitHub github.com modified +186 −0
  • Added Allow: /marketplace/actions/download* for *
  • Added Allow: /topics/comments* for *
  • Added Allow: /topics/download* for *
  • Added Disallow: /marketplace/actions/download*&author=* for *
  • Added Disallow: /marketplace/actions/download*&plan=* for *
  • Added Disallow: /marketplace/actions/download*&q=* for *
  • … 45 more change(s)
view diff → snapshot
Adidas adidas.com modified +1 −4
  • Removed sitemap https://www.adidas.com/qa/ar/sitemap
  • Removed sitemap https://www.adidas.com/qa/en/sitemap
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
JetBlue Airways jetblue.com modified +1 −4
  • Added sitemap https://legacycms.jetblue.com/public/.rest/sitemap/v1/seo/home/sitemap.xml
  • Removed sitemap https://www.globalsiteseo.com/jetblue.GSM/jetblue.gsm.index.en_super.xml
  • Removed sitemap https://www.globalsiteseo.com/jetblue.GSM/jetblue.gsm.index.es_super.xml
  • Removed sitemap https://www.globalsiteseo.com/jetblue.GSM/jetblue.gsm.index.fr_super.xml
  • Removed sitemap https://www.jetblue.com/magnoliapublic/.rest/sitemap/v1/seo/home/sitemap.xml
view diff → snapshot
Synopsys synopsys.com modified +5 −0
  • Added sitemap https://www.synopsys.com/content/dam/synopsys/search/resource-sitemap/dam-sitemap.xml
  • Added sitemap https://www.synopsys.com/ja-jp/sitemap.xml
  • Added sitemap https://www.synopsys.com/ko-kr/sitemap.xml
  • Added sitemap https://www.synopsys.com/zh-cn/sitemap.xml
  • Added sitemap https://www.synopsys.com/zh-tw/sitemap.xml
view diff → snapshot
Tripadvisor tripadvisor.com modified +1 −19
  • Added Disallow: /ProductPayNowNewMethod for *
  • Removed Disallow: /AirportFromGeoAjax for *
  • Removed Disallow: /AirportIP for *
  • Removed Disallow: /CheapFlightsController for *
  • Removed Disallow: /CheapFlightsPartnerHandoff for *
  • Removed Disallow: /CheapFlightsPartnerPostHandoff for *
  • … 45 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +30 −24
  • Added Disallow: /nu/nucoin-app for *
  • Added Disallow: /nu/nucoin-app for DatoCmsSearchBot
view diff → snapshot
Figma Community figma.com modified +5 −0
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/business-design-education/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ko-kr/blog/business-design-education/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-adyen-is-rebuilding-design/$ for Claude-User
  • … 5 more change(s)
view diff → snapshot
Adorama adorama.com modified +3 −1
  • Added Disallow: /alc/category/photography/page/ for *
  • Added Disallow: /alc/wp-json/ for *
  • Added Disallow: /nb/* for *
  • Removed Disallow: /upload/*.m3u8 for *
view diff → snapshot
EDF edf.fr modified +86 −40
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • … 18 more change(s)
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot
Darden Restaurants darden.com modified +7 −77
  • Added Disallow: (empty) for *
  • Removed Allow: /core/*.avif for *
  • Removed Allow: /core/*.css$ for *
  • Removed Allow: /core/*.css? for *
  • Removed Allow: /core/*.gif for *
  • Removed Allow: /core/*.jpeg for *
  • … 45 more change(s)
view diff → snapshot