NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5439snapshots
4304changes detected
239this week
All changes 🤖 AI-crawler changes Hiding initial archives
Deere deere.com modified +1 −13
  • Removed Disallow: /asia/en/search/ for *
  • Removed Disallow: /en/articles/ for *
  • Removed Disallow: /en/finance/offers-discounts/index.html for *
  • Removed Disallow: /en/news/all-news/ for *
  • Removed Disallow: /en/news/all-news/$ for *
  • Removed Disallow: /en/news/all-news/index.html for *
  • … 8 more change(s)
view diff → snapshot
Flickr flickr.com modified +5 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
Expeditors International expeditors.com modified +14 −15
  • Added Disallow: /404-page-not-found/ for *
  • Added Disallow: /error/ for *
  • Added Disallow: /search/ for *
  • Removed Disallow: /404-page-not-found for *
  • Removed Disallow: /error for *
  • Removed Disallow: /search for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Slack slack.com modified +0 −3
  • Removed Allow: /*?*&geocode=* for *
  • Removed Allow: /*?geocode=* for *
  • Removed Disallow: /*?* for *
view diff → snapshot
Figma Community figma.com modified +6 −3
  • 🤖 AI Added Allow: /blog/how-season-5-merch-came-to-life/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/start-anywhere/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/connecting-figma-and-weave/$ for CHATGPT-User
  • … 30 more change(s)
view diff → snapshot
DSV dsv.com removed +0 −34
  • robots.txt is no longer served (HTTP 404)
Gilead Sciences gilead.com modified +2 −0
  • Added Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-methodological-note-2025.pdf for *
  • Added Disallow: /-/media/gileadspain/pdfs/about/transparency/2025/gilead-sciences-spain-efpia-report-2025.pdf for *
view diff → snapshot
Qonto qonto.com modified +2 −490
  • Removed all rules for user-agent ‘AITCSRobot/1.1’
  • Removed all rules for user-agent ‘Alexibot’
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • Removed all rules for user-agent ‘Aqua_Products’
  • Removed all rules for user-agent ‘Arachnophilia’
  • Removed all rules for user-agent ‘ASpider/0.09’
  • … 45 more change(s)
view diff → snapshot
Al Jazeera aljazeera.com modified +1 −1
  • Added Disallow: /*/liveblog/*?*update=* for *
  • Removed Disallow: /*/liveblog/2026/6/*?*update=* for *
view diff → snapshot
Figma Community figma.com modified +30 −0
  • 🤖 AI Added Allow: /de-de/blog/4-new-ways-to-go-from-idea-to-product-with-ai-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/agent-custom-tools-context-skills/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/code-on-the-figma-canvas/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/config-2026-recap/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/figma-make-now-on-your-local-code/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/introducing-figma-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Kimberly-Clark kimberly-clark.com modified +90 −20
  • Added rules for user-agent ‘*’
  • 🤖 AI Blocked anthropic-ai (Anthropic) entirely
  • Added rules for user-agent ‘Applebot’
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • … 22 more change(s)
view diff → snapshot
ACS Group grupoacs.com modified +2 −1
  • Added Disallow: /accionistas-e-inversores-nuevo/* for *
  • Added Disallow: /shareholders-investors-new/* for *
  • Removed Disallow: (empty) for *
view diff → snapshot