NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5433snapshots
4298changes detected
233this week
All changes 🤖 AI-crawler changes Hiding initial archives
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Vimeo vimeo.com modified +357 −0
  • Added Allow: /*/*?%3Bfe= for *
  • Added Allow: /*/*?%3Bfl= for *
  • Added Allow: /*/*?*&%3Bfe= for *
  • Added Allow: /*/*?*&%3Bfl= for *
  • Added Allow: /*/*?*&kref= for *
  • Added Allow: /*/*?*&kuid= for *
  • … 45 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-us-wp-why-ai-security-is-not-an-endpoint-network-or-firewall-problem.pdf for *
view diff → snapshot
Etsy etsy.com modified +60 −0
  • Added Disallow: /appeals/ for *
  • Added Disallow: /at/appeals/ for *
  • Added Disallow: /at/r/group/ for *
  • Added Disallow: /au/appeals/ for *
  • Added Disallow: /au/r/group/ for *
  • Added Disallow: /be/appeals/ for *
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/ai-fluency-isnt-the-finish-line/$ for PerplexityBot
view diff → snapshot
Scale AI scale.com modified +0 −1
  • Removed sitemap https://scale.com/enterprise/prebuilt-applications/sitemap.xml
view diff → snapshot
Ford Motor ford.com modified +10 −6
  • Added sitemap https://www.ford.com/inventory/llms.txt
  • Added sitemap https://www.ford.com/llms.txt
  • Removed sitemap https://www.ford.com/href-sitemap-en-us.xml.
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-us-wp-agent-integrity-framework.pdf for *
view diff → snapshot