NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Showing initial archives
Tesla tesla.com modified +4 −0
  • Added Allow: /sitemap.xml for *
  • Added sitemap https://www.tesla.com/sitemap.xml
view diff → snapshot
Figma Community figma.com modified +4 −0
  • 🤖 AI Added Allow: /resource-library/ai-design-assistant-tools/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/best-ai-video-generators/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/how-to-make-a-cinemagraph/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/sizzle-reel/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-design-assistant-tools/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /resource-library/best-ai-video-generators/$ for Claude-SearchBot
  • … 14 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /de-de/blog/increased-ipo-price-range/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/london-office-expansion/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ko-kr/blog/london-office-expansion/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
General Motors gm.com modified +133 −53
  • Added rules for user-agent ‘AdobeEdgeOptimize/1.0’
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Claude-SearchBot (Anthropic)
  • Added rules for user-agent ‘Googlebot-Mobile’
  • Added rules for user-agent ‘Googlebot-PageRenderer’
  • Added rules for user-agent ‘GoogleMessages’
  • … 45 more change(s)
view diff → snapshot
Microsoft microsoft.com modified +13 −1
  • Added Allow: /en/microsoft-365-copilot/microsoft-copilot-studio for *
  • Added Allow: /en/microsoft-365/business for *
  • Added Allow: /en/microsoft-365/business/microsoft-365-administration for *
  • Added Allow: /en/microsoft-365/clipchamp for *
  • Added Allow: /en/microsoft-365/excel for *
  • Added Allow: /en/microsoft-365/microsoft-office for *
  • … 5 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Fox News foxnews.com modified +1 −3
  • Removed Disallow: /api/article-search for *
  • Removed Disallow: /xid for *
  • Added sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
GitHub github.com modified +56 −0
  • Added Allow: /marketplace/*?tab=readme$ for *
  • Added Allow: /marketplace/*?tab=transparency$ for *
  • Added Disallow: /marketplace/*/*?tab=readme$ for *
  • Added Disallow: /marketplace/*/*?tab=transparency$ for *
  • Added Disallow: /marketplace/*?*?tab=readme$ for *
  • Added Disallow: /marketplace/*?*?tab=transparency$ for *
  • … 45 more change(s)
view diff → snapshot
Lennar lennar.com modified +9 −9
  • Added Disallow: /new-homes/*/amenities for *
  • Added Disallow: /new-homes/*/brochure for *
  • Added Disallow: /new-homes/*/faq for *
  • Added Disallow: /new-homes/*/included for *
  • Added Disallow: /new-homes/*/nearby-places for *
  • Added Disallow: /new-homes/*/nearby-schools for *
  • … 12 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Threads threads.net modified +1 −0
  • Added sitemap https://www.threads.com/sitemap/threads_question_posts_sitemap.xml.gz
view diff → snapshot
Labcorp labcorp.com modified +9 −1
  • Added Disallow: /*?*_ga= for *
  • Added Disallow: /*?*filter= for *
  • Added Disallow: /*?*page= for *
  • Added Disallow: /*?*query= for *
  • Added Disallow: /*?*search= for *
  • Added Disallow: /*?*sort= for *
  • … 2 more change(s)
view diff → snapshot