NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Showing initial archives
GitHub github.com modified +18 −1
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Removed rules for AI crawler anthropic-ai (Anthropic)
  • Added Allow: /enterprise/contact/data-residency?* for *
  • … 34 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
Varonis varonis.com modified +1 −1
  • Added sitemap https://varonis-mcp.webflow.io/sitemap.xml
  • Removed sitemap https://webflow-www.varonis.com/sitemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
DuPont dupont.com removed +0 −294
  • robots.txt is no longer served (HTTP 404)
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Delta Air Lines delta.com modified +0 −12
  • Removed Allow: /app_links/join_skymiles/index.jsp for *
  • Removed Allow: /awards/skymark/landing/index.jsp for *
  • Removed Allow: /cns/travel/flight_notification/index.jsp for *
  • Removed Allow: /components/help/apps/booking/fare_class.jsp for *
  • Removed Allow: /components/popups/skymark/terms.jsp for *
  • Removed Allow: /deltareservemqm/mbrLanding.action for *
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /blog/in-conversation-dylan-field-and-david-friedberg/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/hello-kitty-and-friends-in-figma/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
SerpApi serpapi.com modified +3 −0
  • Added sitemap https://serpapi.com/blog/sitemap.xml
  • Added sitemap https://serpapi.com/sitemap.xml
view diff → snapshot
Varonis varonis.com modified +1 −1
  • Added sitemap https://webflow-www.varonis.com/sitemap.xml
  • Removed sitemap https://varonis-mcp.webflow.io/sitemap.xml
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/threat-reports/pfpt-us-rt-email-without-humans.pdf for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot