NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Showing initial archives
Mondelez International mondelezinternational.com modified +55 −0
  • Added rules for user-agent ‘bingbot’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Google-Extended (Google)
  • … 5 more change(s)
view diff → snapshot
Canva canva.com modified +4 −0
  • 🤖 AI Added Allow: /tr_tr/grafik/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/sema/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/yapma/ for anthropic-ai
  • 🤖 AI Added Allow: /tr_tr/grafik/ for Applebot-Extended
  • 🤖 AI Added Allow: /tr_tr/ozellikler/ for Applebot-Extended
  • … 18 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +7 −2
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /members/*/mem for *
  • Added Disallow: /providers/*/duals for *
  • Added Disallow: /providers/*/medicaid for *
  • Added Disallow: /providers/*/medicare for *
  • Removed Disallow: (empty) for *
  • … 1 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/white-papers/pfpt-wp-jp-votc-2026.pdf for *
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +2 −7
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /members/*/mem for *
  • Removed Disallow: /providers/*/duals for *
  • Removed Disallow: /providers/*/medicaid for *
  • Removed Disallow: /providers/*/medicare for *
  • … 1 more change(s)
view diff → snapshot
JBS jbs.com.br initial
  • First snapshot of robots.txt archived
snapshot
Nubank nubank.com.br modified +18 −12
  • Added Disallow: /nu-empresas-sebrae-conteudos-v2 for *
  • Added Disallow: /nu-empresas-sebrae-conteudos-v2 for DatoCmsSearchBot
view diff → snapshot
Indeed indeed.com modified +2 −0
  • Added Disallow: /promo/ for *
  • 🤖 AI Added Disallow: /promo/ for ChatGPT-User
  • 🤖 AI Added Disallow: /promo/ for Claude-SearchBot
  • 🤖 AI Added Disallow: /promo/ for Claude-User
  • 🤖 AI Added Disallow: /promo/ for DuckAssistBot
  • Added Disallow: /promo/ for DuckDuckBot
  • … 16 more change(s)
view diff → snapshot
Figma Community figma.com modified +0 −67
  • 🤖 AI Removed Allow: /de-de/legal/copyright-and-ip-policy/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/core-features/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/data-portability-and-deletion/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/impressum/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /de-de/legal/shared-responsibility-security-model/$ for CHATGPT-User
  • 🤖 AI Removed Allow: /fr-fr/legal/copyright-and-ip-policy/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Pinterest pinterest.com modified +66 −62
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261206_second_advent_sunday_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261207_pearl_harbor_remembrance_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261210_hanukkah_day_6_rosh_chodesh_tevet_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261210_human_rights_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261210_international_human_rights_day_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/shopping_sitemap_www.pinterest.com.xml
view diff → snapshot
Instagram instagram.com modified +77 −21
  • 🤖 AI Unblocked Amazonbot (Amazon)
  • 🤖 AI Added Allow: /places/c/ for Amazonbot
  • 🤖 AI Added Disallow: /accounts/login/*?next= for Amazonbot
  • 🤖 AI Added Disallow: /ajax/ for Amazonbot
  • 🤖 AI Added Disallow: /client_error/ for Amazonbot
  • 🤖 AI Added Disallow: /direct/ for Amazonbot
  • … 45 more change(s)
view diff → snapshot
Zebra Technologies zebra.com modified +216 −215
  • Added Allow: /jp/ja/support-downloads/mobile-computers/handheld/mc20.html for *
  • Added Disallow: /0003/zebra-it-security-video-portfolio-compliance-en-us.mp4 for *
  • Added Disallow: /3600-series-reference-components.html for *
  • Added Disallow: /5x5-end-user/* for *
  • Added Disallow: /5x5-partner/* for *
  • Added Disallow: /LabelsHero.html for *
  • … 45 more change(s)
view diff → snapshot
NetApp netapp.com modified +82 −27
  • 🤖 AI Added rules for AI crawler Amazonbot (Amazon)
  • 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
  • 🤖 AI Added rules for AI crawler CCBot (Common Crawl)
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-Web (Anthropic)
  • 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
  • … 30 more change(s)
view diff → snapshot