NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Toronto Star thestar.com modified +36 −18
  • Blocked user-agent ‘KeenableBot’ entirely
  • Blocked user-agent ‘recraft-image-dataset’ entirely
  • Blocked user-agent ‘ScryBot’ entirely
  • Blocked user-agent ‘SofyaBot’ entirely
  • Blocked user-agent ‘TalarionFrontierCrawler’ entirely
  • Blocked user-agent ‘YiBot’ entirely
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /type-scale-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /type-scale-generator/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /type-scale-generator/$ for Claude-User
  • 🤖 AI Added Allow: /type-scale-generator/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /type-scale-generator/$ for PerplexityBot
view diff → snapshot
Pinterest pinterest.com modified +64 −58
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_christmas_season_begins_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_hanukkah_day_2_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_saint_nicholas_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_st_nicholas_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261207_hanukkah_day_3_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261207_international_civil_aviation_day_www.pinterest.com.xml
view diff → snapshot
Synopsys synopsys.com modified +1 −1
  • Added sitemap https://www.synopsys.com/dam-sitemap.xml
  • Removed sitemap https://www.synopsys.com/content/dam/synopsys/search/resource-sitemap/dam-sitemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +7 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /members/*/mem for *
  • Added Disallow: /providers/*/duals for *
  • Added Disallow: /providers/*/medicaid for *
  • Added Disallow: /providers/*/medicare for *
  • Removed Disallow: (empty) for *
  • … 1 more change(s)
view diff → snapshot
Progressive progressive.com modified +4 −1
  • Added Allow: /lp/motorcycle/ for *
  • Added Disallow: /*.asp for *
  • Added Disallow: /*lp/ for *
  • Added Disallow: /agent/local-agent/ for *
  • Removed Disallow: /lp/ for *
view diff → snapshot
Toronto Star thestar.com modified +35 −14
  • Blocked user-agent ‘AgentDataBot’ entirely
  • Blocked user-agent ‘BeansLLM-CorpusBot’ entirely
  • Blocked user-agent ‘BixelBot’ entirely
  • Blocked user-agent ‘DesearchBot’ entirely
  • Blocked user-agent ‘Hunyuan’ entirely
  • Blocked user-agent ‘ImageBot’ entirely
  • … 1 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for Claude-User
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /ja-jp/reports/2026-ai-report/$ for PerplexityBot
view diff → snapshot
BBC bbc.com modified +6 −4
  • 🤖 AI Added Allow: /storyworks for OAI-SearchBot
  • 🤖 AI Added Allow: /storyworks/ for OAI-SearchBot
view diff → snapshot
Iceland (Government.is) government.is modified +0 −5
  • Removed Disallow: /page/e0a344e1-b8e8-4f99-873d-9f8cc3d3cb0c for *
  • Removed Disallow: /topics/operations-and-state-assets/policy-on-public-services$ for *
  • Removed Disallow: /topics/operations-and-state-assets/policy-on-public-services/$ for *
  • Removed Disallow: /topics/operations-and-state-assets/policy-on-public-services/? for *
  • Removed Disallow: /topics/operations-and-state-assets/policy-on-public-services? for *
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −7
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /members/*/mem for *
  • Removed Disallow: /providers/*/duals for *
  • Removed Disallow: /providers/*/medicaid for *
  • Removed Disallow: /providers/*/medicare for *
  • … 1 more change(s)
view diff → snapshot
Arrow Electronics arrow.com created +13 −0
  • 🤖 AI Added Allow: /globalecs/* for anthropic-ai
  • 🤖 AI Added Allow: /globalecs/* for CCBot
  • 🤖 AI Added Allow: /globalecs/* for ChatGPT-User
  • 🤖 AI Added Allow: /globalecs/* for ClaudeBot
  • 🤖 AI Added Allow: /globalecs/* for Google-Extended
  • 🤖 AI Added Allow: /globalecs/* for GPTBot
  • … 1 more change(s)
view diff → snapshot
Mondelez International mondelezinternational.com modified +55 −0
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • … 5 more change(s)
view diff → snapshot
Canva canva.com modified +4 −0
  • 🤖 AI Added Allow: /tr_tr/grafik/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/sema/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/yapma/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/grafik/ for Applebot-Extended
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for Applebot-Extended
  • … 18 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot