NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2601snapshots
1428changes detected
255this week
All changes 🤖 AI-crawler changes Showing initial archives
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Arrow Electronics arrow.com modified +0 −5
  • Removed Disallow: /ko/categories for *
  • Removed Disallow: /ko/manufacturers for *
  • Removed Disallow: /ko/products/ for *
  • Removed Disallow: /ko/resources/reference-designs for *
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +17 −0
  • 🤖 AI Added Allow: /customers/duotone-saves-hours/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/4-ways-were-using-our-mcp-server-at-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/figma-zu-code-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/mockup-zu-prototyp-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/sketch-zu-figma-konverter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-ways-were-using-our-mcp-server-at-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
USA Today usatoday.com modified +6 −0
  • Added Allow: /money/money-management/ for ChatGPT Operator
  • 🤖 AI Added Allow: /money/money-management/ for ChatGPT-User
  • 🤖 AI Added Allow: /money/money-management/ for Google-Extended
  • 🤖 AI Added Allow: /money/money-management/ for GPTBot
  • 🤖 AI Added Allow: /money/money-management/ for OAI-SearchBot
  • 🤖 AI Added Allow: /money/money-management/ for PerplexityBot
view diff → snapshot
The Washington Post washingtonpost.com modified +3 −1
  • 🤖 AI Blocked PerplexityBot (Perplexity) entirely
  • 🤖 AI Added Allow: /advertising/ for PerplexityBot
  • 🤖 AI Added Allow: /creativegroup/ for PerplexityBot
  • 🤖 AI Removed Allow: / for PerplexityBot
view diff → snapshot
Shell shell.com modified +2 −0
  • Added Disallow: /business-customers/chemicals/home-care/fabric-care-brochure.html for *
  • Added Disallow: /business-customers/chemicals/home-care/fabric-care-brochure/ for *
view diff → snapshot
Ralph Lauren ralphlauren.com modified +2 −28
  • Added Disallow: */r/*/* for *
  • Added Disallow: */r/*_* for *
  • Removed Allow: /sale-new-to-sale-men-cg/r/polo-ralph-lauren for *
  • Removed Disallow: */r/*_big-and-tall for *
  • Removed Disallow: */r/*_black for *
  • Removed Disallow: */r/*_blue for *
  • … 24 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
General Mills generalmills.com modified +9 −2
  • Added Disallow: /*.aspx for *
  • Added Disallow: /*p=1 for *
  • Added Disallow: /.well-known/apple-app-site-association for *
  • Added Disallow: /apple-app-site-association for *
  • Added Disallow: /search for *
  • Added Disallow: /shared/ for *
  • … 4 more change(s)
view diff → snapshot