NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Showing initial archives
Bild bild.de modified +0 −16
  • Removed Disallow: /casinos/go/ for *
  • Removed Disallow: /dating/go/ for *
  • Removed Disallow: /energieloesungen/go/ for *
  • Removed Disallow: /energieloesungen/social/go/ for *
  • Removed Disallow: /energieloesungen/vergleich/go/ for *
  • Removed Disallow: /games/go/ for *
  • … 10 more change(s)
view diff → snapshot
Westpac westpac.com.au modified +17 −27
  • Added Allow: / for *
  • Added Disallow: /test-aem-1817/ for *
  • Added Disallow: /test/ for *
  • Removed Disallow: /Forms/ for *
  • Removed Disallow: /business-banking/business-banking-online-sign-in-out/signin/ for *
  • Removed Disallow: /business-banking/merchant-services/mobile-payments/mobilepayway/ for *
  • … 18 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Samsung Electronics samsung.com modified +8 −3
  • Added Allow: /ca/multistore/offers/back-to-school/ for *
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for *
  • Added Allow: /ca/multistore/offers/back-to-school/ for AdsBot-Google
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for AdsBot-Google
  • 🤖 AI Added Allow: /ca/multistore/offers/back-to-school/ for ChatGPT-User
  • 🤖 AI Added Allow: /ca_fr/multistore/offers/back-to-school/ for ChatGPT-User
  • … 17 more change(s)
view diff → snapshot
Anysphere (Cursor) cursor.com modified +0 −13
  • Removed Allow: /docs-static/apple-touch-icon* for *
  • Removed Allow: /docs-static/favicon* for *
  • Removed Allow: /docs-static/icon-* for *
  • Removed Allow: /marketing-static/apple-touch-icon* for *
  • Removed Allow: /marketing-static/favicon* for *
  • Removed Allow: /marketing-static/icon-* for *
  • … 2 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +7 −0
  • Added Disallow: /sites/default/files/misc/pfpt-br-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-de-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-es-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-fr-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-it-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-jp-rt-2026-ai-era-ransomware.pdf for *
  • … 1 more change(s)
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://www.thetradedesk.com/sitemap.xml
  • Removed sitemap https://careers.thetradedesk.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Vox vox.com modified +0 −100
  • Removed Allow: /strategist for A6-Indexer
  • Removed Allow: /strategist for AhrefsBot
  • 🤖 AI Removed Allow: /strategist for AI2Bot
  • Removed Allow: /strategist for AI2Bot-DeepResearchEval
  • Removed Allow: /strategist for Ai2Bot-Dolma
  • Removed Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Realtor realtor.com modified +0 −2
  • Removed Disallow: /realestateandhomes-search/*/price- for *
  • Removed Disallow: /realestateandhomes-search/*/price-na-*/pg- for *
view diff → snapshot
Nubank nubank.com.br modified +60 −42
  • Added Disallow: /home for *
  • Added Disallow: /home2 for *
  • Added Disallow: /home3 for *
  • Added Disallow: /home for DatoCmsSearchBot
  • Added Disallow: /home2 for DatoCmsSearchBot
  • Added Disallow: /home3 for DatoCmsSearchBot
view diff → snapshot