NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
Heineken heineken.com modified +10 −0
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
  • … 2 more change(s)
view diff → snapshot
Arrow Electronics arrow.com modified +1 −3
  • 🤖 AI Unblocked GPTBot (OpenAI)
  • 🤖 AI Added Allow: / for GPTBot
  • 🤖 AI Removed Allow: /ais/* for GPTBot
  • 🤖 AI Removed Allow: /globalecs/* for GPTBot
view diff → snapshot
Mastodon mastodon.social modified +21 −1
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘Ai2Bot-Dolma’ entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • … 14 more change(s)
view diff → snapshot
Europe PMC europepmc.org modified +70 −1
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘BorderxBot’ entirely
  • … 19 more change(s)
view diff → snapshot
Figma Community figma.com modified +21 −0
  • 🤖 AI Added Allow: /de-de/blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Guyana (Office of the President) op.gov.gy modified +130 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 27 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for Claude-User
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/6-ways-teams-are-experimenting-with-code-layers/$ for PerplexityBot
view diff → snapshot
Figma Community figma.com modified +46 −0
  • 🤖 AI Added Allow: /de-de/blog/4-team-rituals-that-drive-ai-adoption/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-together-with-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/measuring-the-impact-of-ai/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/downloads/chrome-extension/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/4-team-rituals-that-drive-ai-adoption/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/design-together-with-code-layers/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −49
  • Removed all rules for user-agent ‘*’
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • 🤖 AI Unblocked Google-Extended (Google) - rules removed
  • … 4 more change(s)
view diff → snapshot
Costa Rica (Presidencia) presidencia.go.cr modified +128 −70
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 45 more change(s)
view diff → snapshot
The Telegraph telegraph.co.uk modified +69 −7
  • Removed Disallow: /promotions/emails/ for *
  • Removed Disallow: /sponsored/travel/msc-cruises/ for *
  • 🤖 AI Added Allow: /bah-avis/usa-sports-road-trip-map for AI2Bot
  • 🤖 AI Added Allow: /bespoke-content/ for AI2Bot
  • 🤖 AI Added Allow: /betting/ for AI2Bot
  • 🤖 AI Added Allow: /brand-usa for AI2Bot
  • … 45 more change(s)
view diff → snapshot
Figma Community figma.com modified +27 −0
  • 🤖 AI Added Allow: /blog/how-to-put-your-design-system-to-work-with-the-figma-agent/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-specs/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/sightlines-issue-no3/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/figma-deepens-investment-in-india/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/introducing-figma-specs/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
NBC News nbcnews.com modified +9 −8
  • 🤖 AI Added Allow: /information/nbc-news-info/about-nbc-news-digital-n1232178 for Meta-WebIndexer
  • 🤖 AI Added Allow: /information/nbc-news-info/closed-captioning-n1307063 for Meta-WebIndexer
  • 🤖 AI Added Allow: /information/nbc-news-info/contact-us-n1232521 for Meta-WebIndexer
  • 🤖 AI Added Allow: /news/us-news/nbc-news-digital-editors-n893846 for Meta-WebIndexer
  • 🤖 AI Added Allow: /select for Meta-WebIndexer
  • 🤖 AI Added Allow: /select/ for Meta-WebIndexer
  • … 28 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for Claude-User
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/lufthansa-figma-make/$ for PerplexityBot
view diff → snapshot
Sony Music sonymusic.com modified +128 −6
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 29 more change(s)
view diff → snapshot
NBC News nbcnews.com modified +25 −0
  • 🤖 AI Blocked meta-externalfetcher (Meta) entirely
  • 🤖 AI Blocked Meta-WebIndexer (Meta) entirely
  • Blocked user-agent ‘meta-webindexer/1.1’ entirely
view diff → snapshot
Figma Community figma.com modified +0 −358
  • 🤖 AI Removed Allow: /de-de/vorlagen/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/vorlagen/affinitatsdiagramm-beispiele/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/vorlagen/agile-roadmap-vorlage/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/vorlagen/angebots-und-nachfragekurve/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/vorlagen/beispiel-kanban-board/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/vorlagen/betriebsablaufe/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Sophos sophos.com modified +7 −1
  • Added rules for user-agent ‘Amzn-SearchBot’
  • Added rules for user-agent ‘Amzn-User’
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: / for Amazonbot
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for Claude-User
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /blog/source-material-ben-ganz/$ for PerplexityBot
view diff → snapshot
Daily Mail dailymail.co.uk modified +2 −1
  • Added Allow: /llms.txt for AhrefsBot
  • 🤖 AI Added Allow: /llms.txt for AI2Bot
  • Added Allow: /llms.txt for Ai2Bot-Dolma
  • Added Allow: /llms.txt for amazon-kendra-
  • 🤖 AI Added Allow: /llms.txt for Amazonbot
  • 🤖 AI Added Allow: /llms.txt for anthropic-ai
  • … 45 more change(s)
view diff → snapshot
Cencora cencora.com modified +14 −2
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • Added rules for user-agent ‘Applebot’
  • Added rules for user-agent ‘Bingbot’
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler cohere-ai (Cohere)
  • 🤖 AI Added rules for AI crawler Diffbot (Diffbot)
  • … 6 more change(s)
view diff → snapshot
Iceland (Government.is) government.is modified +130 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 27 more change(s)
view diff → snapshot
Lithia Motors lithiamotors.com modified +70 −1
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • Blocked user-agent ‘Baiduspider’ entirely
  • Blocked user-agent ‘BorderxBot’ entirely
  • … 19 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +49 −0
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked CCBot (Common Crawl) entirely
  • 🤖 AI Blocked ClaudeBot (Anthropic) entirely
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • … 4 more change(s)
view diff → snapshot
Expeditors International expeditors.com modified +130 −0
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • Blocked user-agent ‘atlassian-bot’ entirely
  • Blocked user-agent ‘AwarioRssBot’ entirely
  • Blocked user-agent ‘AwarioSmartBot’ entirely
  • … 27 more change(s)
view diff → snapshot
Instagram instagram.com modified +18 −66
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Removed Allow: /places/c/ for Amazonbot
  • 🤖 AI Removed Disallow: /accounts/login/*?next= for Amazonbot
  • 🤖 AI Removed Disallow: /ajax/ for Amazonbot
  • 🤖 AI Removed Disallow: /client_error/ for Amazonbot
  • 🤖 AI Removed Disallow: /direct/ for Amazonbot
  • … 45 more change(s)
view diff → snapshot