NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
Samsung Electronics samsung.com modified +8 −3
  • Added Allow: /ca/multistore/offers/back-to-school/ for *
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for *
  • Added Allow: /ca/multistore/offers/back-to-school/ for AdsBot-Google
  • Added Allow: /ca_fr/multistore/offers/back-to-school/ for AdsBot-Google
  • 🤖 AI Added Allow: /ca/multistore/offers/back-to-school/ for ChatGPT-User
  • 🤖 AI Added Allow: /ca_fr/multistore/offers/back-to-school/ for ChatGPT-User
  • … 17 more change(s)
view diff → snapshot
Anysphere (Cursor) cursor.com modified +0 −13
  • Removed Allow: /docs-static/apple-touch-icon* for *
  • Removed Allow: /docs-static/favicon* for *
  • Removed Allow: /docs-static/icon-* for *
  • Removed Allow: /marketing-static/apple-touch-icon* for *
  • Removed Allow: /marketing-static/favicon* for *
  • Removed Allow: /marketing-static/icon-* for *
  • … 2 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +7 −0
  • Added Disallow: /sites/default/files/misc/pfpt-br-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-de-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-es-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-fr-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-it-rt-2026-ai-era-ransomware.pdf for *
  • Added Disallow: /sites/default/files/misc/pfpt-jp-rt-2026-ai-era-ransomware.pdf for *
  • … 1 more change(s)
view diff → snapshot
The Trade Desk thetradedesk.com modified +1 −1
  • Added sitemap https://www.thetradedesk.com/sitemap.xml
  • Removed sitemap https://careers.thetradedesk.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +14 −0
  • 🤖 AI Added Allow: /de-de/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/how-to-move-fast-toward-the-right-thing/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/try-these-10-skills-and-show-off-your-own/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Vox vox.com modified +0 −100
  • Removed Allow: /strategist for A6-Indexer
  • Removed Allow: /strategist for AhrefsBot
  • 🤖 AI Removed Allow: /strategist for AI2Bot
  • Removed Allow: /strategist for AI2Bot-DeepResearchEval
  • Removed Allow: /strategist for Ai2Bot-Dolma
  • Removed Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Realtor realtor.com modified +0 −2
  • Removed Disallow: /realestateandhomes-search/*/price- for *
  • Removed Disallow: /realestateandhomes-search/*/price-na-*/pg- for *
view diff → snapshot
Nubank nubank.com.br modified +60 −42
  • Added Disallow: /home for *
  • Added Disallow: /home2 for *
  • Added Disallow: /home3 for *
  • Added Disallow: /home for DatoCmsSearchBot
  • Added Disallow: /home2 for DatoCmsSearchBot
  • Added Disallow: /home3 for DatoCmsSearchBot
view diff → snapshot
Thyssenkrupp thyssenkrupp.com modified +9 −1
  • Added Disallow: *UCPthyssenkruppAG* for *
  • Added Disallow: /_api for *
  • Added Disallow: /_media/ for *
  • Added Disallow: /assets.files/ for *
  • Added Disallow: /de/home-2604 for *
  • Added Disallow: /de/newsroom/pressemeldungen/press-release- for *
  • … 4 more change(s)
view diff → snapshot
Daily Mail dailymail.co.uk modified +8 −17
  • Added sitemap https://www.dailymail.com/video-new-sitemap.xml
  • Removed sitemap https://www.dailymail.com/sitemap-authors.xml
  • Removed sitemap https://www.dailymail.com/sitemap-channels.xml
  • Removed sitemap https://www.dailymail.com/sitemap-columnist.xml
  • Removed sitemap https://www.dailymail.com/sitemap-facts-year~2021.xml
  • Removed sitemap https://www.dailymail.com/sitemap-facts-year~2022.xml
  • … 5 more change(s)
view diff → snapshot
ESET eset.com modified +19 −0
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • Blocked user-agent ‘BacklinkBot’ entirely
  • Blocked user-agent ‘Brandwatch’ entirely
  • Blocked user-agent ‘CognitiveSEO’ entirely
  • Blocked user-agent ‘DataForSeoBot’ entirely
  • … 13 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for CHATGPT-User
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /fr-fr/blog/the-benefits-of-code-connect-in-mcp/$ for Claude-User
  • 🤖 AI Added Allow: /uk-tax-strategy/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Vox vox.com modified +100 −0
  • Added Allow: /strategist for A6-Indexer
  • Added Allow: /strategist for AhrefsBot
  • 🤖 AI Added Allow: /strategist for AI2Bot
  • Added Allow: /strategist for AI2Bot-DeepResearchEval
  • Added Allow: /strategist for Ai2Bot-Dolma
  • Added Allow: /strategist for aiHitBot
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Pinterest pinterest.com modified +12 −13
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideas_hub_trending_ideas_terms_sitemap_www.pinterest.com.xml
view diff → snapshot