NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +18 −0
  • 🤖 AI Added Allow: /de-de/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
AT&T att.com modified +46 −48
  • Added Allow: /.well-known/ai-catalog.json for *
  • Added Allow: /.well-known/api-catalog.txt for *
  • Added Allow: /ai.txt for *
  • Added Allow: /llms-full.txt for *
  • Added Allow: /llms.txt for *
  • Added Allow: /openapi.json for *
  • … 19 more change(s)
view diff → snapshot
Twitch twitch.tv modified +35 −11
  • Added Disallow: /activate$ for *
  • Added Disallow: /activate/ for *
  • Added Disallow: /activate? for *
  • Added Disallow: /claim$ for *
  • Added Disallow: /claim/ for *
  • Added Disallow: /claim? for *
  • … 40 more change(s)
view diff → snapshot
OpenAlex openalex.org modified +2 −61
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler meta-externalagent (Meta)
view diff → snapshot
Eastman Chemical eastman.com modified +2 −15
  • Blocked user-agent ‘*’ entirely
  • Removed Disallow: */_vti_bin/ for *
  • Removed Disallow: */product-inquiry for *
  • Removed Disallow: */search-results for *
  • Removed Disallow: /Brands/Eastar/_layouts/ for *
  • Removed Disallow: /Brands/TamiSolveNxG/_layouts/ for *
  • … 6 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −29
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 23 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for PerplexityBot
view diff → snapshot
seoClarity seoclarity.net modified +19 −2
  • Blocked user-agent ‘BrightEdge Crawler/1.0’ entirely
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • Added Disallow: /hubfs/Client%20Stories/Client%20Testimonials.pdf for *
  • Added Disallow: /seo-platform-essentials/confirmation-lead/thank-you for *
  • Removed Allow: /seo-platform-essentials/confirmation-lead/thank-you for *
  • Removed Disallow: /workflow-video-library for *
  • … 1 more change(s)
view diff → snapshot
OpenAlex openalex.org modified +121 −0
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 39 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +106 −106
  • 🤖 AI Removed rules for AI crawler meta-externalfetcher (Meta)
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261126_thanksgiving_www.pinterest.com.xml
view diff → snapshot
Tesla tesla.com modified +0 −4
  • Removed Allow: /sitemap.xml for *
  • Removed sitemap https://www.tesla.com/sitemap.xml
view diff → snapshot
AliExpress aliexpress.com modified +22 −3
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/design-together-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/introducing-figma-specs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /specs/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot
Instagram instagram.com modified +3 −11
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Removed Allow: /places/c/ for Google-Extended
  • 🤖 AI Removed Disallow: /accounts/login/*?next= for Google-Extended
  • 🤖 AI Removed Disallow: /ajax/ for Google-Extended
  • 🤖 AI Removed Disallow: /client_error/ for Google-Extended
  • 🤖 AI Removed Disallow: /direct/ for Google-Extended
  • … 4 more change(s)
view diff → snapshot