NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Showing initial archives
Figma Community figma.com modified +15 −0
  • 🤖 AI Added Allow: /de-de/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/principles-in-motion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/pantone-speaking-the-language-of-color/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/principles-in-motion/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Wired wired.com modified +3 −1
  • Added Allow: /*?*utm_medium=social for *
  • Added Allow: /*?ref= for *
view diff → snapshot
Pinterest pinterest.com modified +39 −41
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_christmas_sitemap_www.pinterest.com.xml
  • Removed sitemap https://www.pinterest.com/v3_sitemaps/ideahub_new_years_eve_sitemap_www.pinterest.com.xml
view diff → snapshot
Rockwell Automation rockwellautomation.com modified +0 −53
  • Removed Disallow: /de-at/docs for *
  • Removed Disallow: /de-ch/docs for *
  • Removed Disallow: /de-de/docs for *
  • Removed Disallow: /en-au/docs for *
  • Removed Disallow: /en-be/docs for *
  • Removed Disallow: /en-ca/docs for *
  • … 39 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
SoundCloud soundcloud.com modified +29 −60
  • Blocked user-agent ‘Brightbot’ entirely
  • 🤖 AI Blocked omgili (Webz.io) entirely
  • Blocked user-agent ‘omgilibot’ entirely
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • … 45 more change(s)
view diff → snapshot
Palo Alto Networks paloaltonetworks.com modified +108 −0
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • 🤖 AI Added rules for AI crawler Claude-User (Anthropic)
  • 🤖 AI Added rules for AI crawler OAI-SearchBot (OpenAI)
  • 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
X (Twitter) x.com modified +39 −21
  • Added Allow: /i/api/* for Bingbot
  • Added Disallow: /*/likes for Bingbot
  • Added Disallow: /*/likes? for Bingbot
  • Added Disallow: /*/media for Bingbot
  • Added Disallow: /*/media? for Bingbot
  • Added Disallow: /*/photo$ for Bingbot
  • … 40 more change(s)
view diff → snapshot
ChatGPT chatgpt.com modified +6 −0
  • Added rules for user-agent ‘AdsBot-Google’
  • Added rules for user-agent ‘AdsBot-Google-Mobile’
view diff → snapshot
SoundCloud soundcloud.com modified +60 −29
  • 🤖 AI Blocked Amazonbot (Amazon) entirely
  • 🤖 AI Blocked Applebot-Extended (Apple) entirely
  • 🤖 AI Blocked Bytespider (ByteDance) entirely
  • 🤖 AI Blocked ChatGPT-User (OpenAI) entirely
  • 🤖 AI Blocked Claude-Web (Anthropic) entirely
  • 🤖 AI Blocked cohere-ai (Cohere) entirely
  • … 45 more change(s)
view diff → snapshot