NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
Varonis varonis.com modified +1 −1
  • Added sitemap https://webflow-www.varonis.com/sitemap.xml
  • Removed sitemap https://varonis-mcp.webflow.io/sitemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +6 −12
  • Removed Disallow: /nu-empresas-sebrae-conteudos-v2 for *
  • Removed Disallow: /nu-empresas-sebrae-conteudos-v2 for DatoCmsSearchBot
view diff → snapshot
Figma Community figma.com modified +8 −3
  • 🤖 AI Added Allow: /es-es/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /pt-br/legal/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/legal/$ for Claude-SearchBot
  • … 19 more change(s)
view diff → snapshot
Bayer bayer.com modified +79 −0
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 9 more change(s)
view diff → snapshot
Varonis varonis.com modified +10 −5
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • Removed Disallow: /*?*hsCacheBuster=* for *
  • Removed Disallow: /*?*hs_preview=* for *
  • Removed Disallow: /_hcms/preview/ for *
  • Removed Disallow: /hs/manage-preferences/ for *
  • … 3 more change(s)
view diff → snapshot
Ahrefs ahrefs.com modified +0 −2
  • Removed Disallow: /academy/draft/* for *
  • Removed Disallow: /draft/* for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +29 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 23 more change(s)
view diff → snapshot
Realtor realtor.com modified +39 −3
  • Removed all rules for user-agent ‘bingbot’
  • Added Allow: /realestateandhomes-search/*/district/*/* for *
  • Added Allow: /realestateandhomes-search/*/district/*/price-na- for *
  • Added Allow: /realestateandhomes-search/*/park/*/* for *
  • Added Allow: /realestateandhomes-search/*/park/*/price-na- for *
  • Added Allow: /realestateandhomes-search/*/schools/*/* for *
  • … 34 more change(s)
view diff → snapshot
Nubank nubank.com.br modified +0 −6
  • Removed Disallow: /shopee for *
  • Removed Disallow: /shopee for DatoCmsSearchBot
view diff → snapshot
Delta Air Lines delta.com modified +5 −6
  • Added Allow: /flightstatus/search for *
  • Added Disallow: /flight-status* for *
  • Added Disallow: /flightstatus/1* for *
  • Removed Allow: /dlsearch/ for *
  • Removed Allow: /egift/eGiftPurchase.action for *
  • Removed Disallow: *flight-status* for *
  • … 2 more change(s)
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-ben-ganz/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Orange orange.com modified +2 −2
  • Added Disallow: /en/contact-email/* for *
  • Added Disallow: /fr/contact-email/* for *
view diff → snapshot
ResearchGate researchgate.net modified +5 −0
  • Blocked user-agent ‘GoogleOther’ entirely
  • Blocked user-agent ‘GoogleOther-Image’ entirely
  • Blocked user-agent ‘GoogleOther-Video’ entirely
view diff → snapshot
Glencore glencore.com modified +0 −4
  • Removed Disallow: /.rest/api/v1/documents/1ab16a3be154f86e1478a8f0e6277f65/D%C3%A9ploiement+de+Coupa%21+_+3+novembre+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/2c4c65ce2237cc9d2efdd214ec1b85df/We+are+Live%21+_+November+3%2C+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/535eca4046bad6d3dec0a5c608299fd0/Inscription+et+Formation+Portail+Fournisseur+Coupa-+8+septembre+2025.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/e33dab157f5c29da07a1873996829885/Informations+Importantes+-+Prochain+Lancement+de+Coupa+-+2+Juillet+2025+%281%29.pdf for *
view diff → snapshot
Zhongji Innolight innolight.com modified +3 −13
  • Added Allow: / for *
  • Removed Disallow: /bin/ for *
  • Removed Disallow: /css/ for *
  • Removed Disallow: /demo/ for *
  • Removed Disallow: /extend/ for *
  • Removed Disallow: /fonts/ for *
  • … 9 more change(s)
view diff → snapshot