NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Showing initial archives
Facebook facebook.com modified +18 −282
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: /ajax/bootloader-endpoint/ for Amazonbot
  • 🤖 AI Removed Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Amazonbot
  • 🤖 AI Removed Allow: /careers/ for Amazonbot
  • 🤖 AI Removed Allow: /safetycheck/ for Amazonbot
  • 🤖 AI Removed Disallow: /*/plugins/* for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Coursera coursera.org modified +94 −190
  • Added Allow: /courses?*&page=1$ for *
  • Added Allow: /courses?*&page=1& for *
  • Added Allow: /courses?*&page=2$ for *
  • Added Allow: /courses?*&page=2& for *
  • Added Allow: /courses?*&page=3$ for *
  • Added Allow: /courses?*&page=3& for *
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Cisco Systems cisco.com modified +0 −19
  • Removed all rules for user-agent ‘Bingbot’
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • Removed all rules for user-agent ‘Google AdsBot’
  • 🤖 AI Removed rules for AI crawler Google-Extended (Google)
  • Removed all rules for user-agent ‘Googlebot’
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +20 −0
  • 🤖 AI Added Allow: /es-es/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /hi-in/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /it-it/agent/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
EPFL epfl.ch modified +0 −130
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked anthropic-ai (Anthropic) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • Removed all rules for user-agent ‘atlassian-bot’
  • Removed all rules for user-agent ‘AwarioRssBot’
  • … 27 more change(s)
view diff → snapshot
Alamy alamy.com modified +3 −0
  • Added Disallow: /fpanalytics/ for *
  • Added Disallow: /purchased-assets/ for *
  • Added Disallow: /search-assistant/ for *
  • Added Disallow: /fpanalytics/ for AdsBot-Google
  • Added Disallow: /purchased-assets/ for AdsBot-Google
  • Added Disallow: /search-assistant/ for AdsBot-Google
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +18 −0
  • 🤖 AI Added Allow: /de-de/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/3-ways-product-designers-use-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/figma-agent-and-design-systems/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
OpenAlex openalex.org modified +2 −61
  • 🤖 AI Removed rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Removed rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Removed rules for AI crawler meta-externalagent (Meta)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/how-eternal-uses-figma-to-build-scale-business/$ for PerplexityBot
view diff → snapshot
seoClarity seoclarity.net modified +19 −2
  • Blocked user-agent ‘BrightEdge Crawler/1.0’ entirely
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • Added Disallow: /hubfs/Client%20Stories/Client%20Testimonials.pdf for *
  • Added Disallow: /seo-platform-essentials/confirmation-lead/thank-you for *
  • Removed Allow: /seo-platform-essentials/confirmation-lead/thank-you for *
  • Removed Disallow: /workflow-video-library for *
  • … 1 more change(s)
view diff → snapshot
OpenAlex openalex.org modified +121 −0
  • 🤖 AI Added rules for AI crawler Applebot-Extended (Apple)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 39 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +106 −106
  • 🤖 AI Removed rules for AI crawler meta-externalfetcher (Meta)
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261126_thanksgiving_www.pinterest.com.xml
view diff → snapshot
AliExpress aliexpress.com modified +22 −3
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Blocked GPTBot (OpenAI) entirely
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +6 −0
  • 🤖 AI Added Allow: /blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/design-together-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/introducing-figma-specs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /specs/$ for CHATGPT-User
  • … 24 more change(s)
view diff → snapshot
Instagram instagram.com modified +3 −11
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Removed Allow: /places/c/ for Google-Extended
  • 🤖 AI Removed Disallow: /accounts/login/*?next= for Google-Extended
  • 🤖 AI Removed Disallow: /ajax/ for Google-Extended
  • 🤖 AI Removed Disallow: /client_error/ for Google-Extended
  • 🤖 AI Removed Disallow: /direct/ for Google-Extended
  • … 4 more change(s)
view diff → snapshot
Facebook facebook.com modified +3 −47
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • 🤖 AI Removed Allow: /ajax/bootloader-endpoint/ for Google-Extended
  • 🤖 AI Removed Allow: /ajax/pagelet/generic.php/PagePostsSectionPagelet for Google-Extended
  • 🤖 AI Removed Allow: /careers/ for Google-Extended
  • 🤖 AI Removed Allow: /safetycheck/ for Google-Extended
  • 🤖 AI Removed Disallow: /*/plugins/* for Google-Extended
  • … 40 more change(s)
view diff → snapshot
CNN cnn.com modified +2 −1
  • Blocked user-agent ‘ExaSearchBot’ entirely
  • Blocked user-agent ‘ExaUser’ entirely
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /id-id/solusi/penghapus-latar-belakang-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /nl-nl/oplossingen/ai-achtergrond-verwijderaar/$ for CHATGPT-User
  • 🤖 AI Added Allow: /th-th/solutions/ai-background-remover/$ for CHATGPT-User
  • 🤖 AI Added Allow: /id-id/solusi/penghapus-latar-belakang-ai/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /nl-nl/oplossingen/ai-achtergrond-verwijderaar/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /th-th/solutions/ai-background-remover/$ for Claude-SearchBot
  • … 9 more change(s)
view diff → snapshot
Academia.edu academia.edu modified +55 −0
  • Added Disallow: /published_today_preview for APIs-Google
  • Added Disallow: /published_today_preview for Applebot
  • Added Disallow: /published_today_preview for ArchitextSpider
  • Added Disallow: /published_today_preview for baiduspider
  • Added Disallow: /published_today_preview for bingbot
  • Added Disallow: /published_today_preview for BingPreview
  • … 45 more change(s)
view diff → snapshot
Toyota toyota.com modified +140 −69
  • Removed Disallow: /dealers* for *
  • 🤖 AI Unblocked Amazonbot (Amazon)
  • 🤖 AI Unblocked anthropic-ai (Anthropic)
  • 🤖 AI Unblocked Applebot-Extended (Apple)
  • 🤖 AI Unblocked Bytespider (ByteDance)
  • 🤖 AI Unblocked CCBot (Common Crawl)
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-manon-macasaet/$ for Claude-User
  • 🤖 AI Added Allow: /customers/how-figma-helps-revolut-deliver-wow-for-customers/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +61 −2
  • 🤖 AI Added Allow: /de-de/loesungen/anzeigen-ersteller/facebook/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-app-icon-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-dateireiniger/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/loesungen/ki-schaltflaechengenerator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/customers/how-adyen-is-rebuilding-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/soluciones/creador-de-anuncios/facebook/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
CarMax carmax.com modified +36 −0
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
IGN ign.com modified +5 −0
  • Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for "008"
  • Added Allow: /articles/z-fold8-vs-previous-samsung-foldables-how-battery-performance-efficiency-and-display-experience-have-evolved for "008"
  • Added Allow: /sponsored/ for *
  • Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for AddSearchBot
  • Added Allow: /articles/z-fold8-vs-previous-samsung-foldables-how-battery-performance-efficiency-and-display-experience-have-evolved for AddSearchBot
  • Added Allow: /articles/how-the-samsung-galaxy-z-fold8-has-been-optimised-for-large-screen-entertainment-multitasking-and-all-day-battery-life for AgentDataBot
  • … 45 more change(s)
view diff → snapshot
EPFL epfl.ch modified +130 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • … 27 more change(s)
view diff → snapshot
DBLP dblp.org modified +66 −165
  • Removed all rules for user-agent ‘AddSearchBot’
  • Removed all rules for user-agent ‘AgentTimes’
  • 🤖 AI Unblocked AI2Bot (Allen Institute for AI) - rules removed
  • Removed all rules for user-agent ‘AI2Bot-DeepResearchEval’
  • Removed all rules for user-agent ‘Ai2Bot-Dolma’
  • Removed all rules for user-agent ‘aiHitBot’
  • … 45 more change(s)
view diff → snapshot