NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Lowe's lowes.com modified +8 −14
  • Added Disallow: */1-top-rated/* for *
  • Added Disallow: */2-top-rated/* for *
  • Added Disallow: */3-top-rated/* for *
  • Added Disallow: */4-top-rated/* for *
  • Removed Disallow: /*,5007197 for *
  • Removed Disallow: /*5005558 for *
  • … 9 more change(s)
view diff → snapshot
Figma Community figma.com modified +7 −0
  • 🤖 AI Added Allow: /blog/source-material-aitana-pico/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/csv-to-chart-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/csv-to-dashboard-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/image-to-icon-set-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/logo-to-brand-kit-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/prd-to-prototype-generator/$ for CHATGPT-User
  • … 29 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −137
  • robots.txt is no longer served (HTTP 404)
Coursera coursera.org modified +4 −11
  • Added rules for user-agent ‘AdsBot-Google’
  • Added Disallow: /5JKLVK/ for *
  • Removed Allow: /courses?*&page=1$ for *
  • Removed Allow: /courses?*&page=1& for *
  • Removed Allow: /courses?*&page=2$ for *
  • Removed Allow: /courses?*&page=2& for *
  • … 18 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +21 −17
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261228_substitute_bank_holiday_for_boxing_day_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_december_31_bank_holiday_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_international_new_year_s_eve_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261231_new_year_holiday_www.pinterest.com.xml
view diff → snapshot
Penguin Random House penguinrandomhouse.com modified +11 −1
  • Unblocked user-agent ‘*’
  • Added Allow: /wp-admin/admin-ajax.php for *
  • Added Allow: /wp-includes/css/dist/block-library/style.min.css for *
  • Added Disallow: /interactive/reading-preference for *
  • Added Disallow: /prh-internal-news/ for *
  • Added Disallow: /search/ for *
  • … 5 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Yandex yandex.ru modified +0 −190
  • Removed Allow: /blogs$ for *
  • Removed Allow: /blogs/$ for *
  • Removed Allow: /blogs/pad$ for *
  • Removed Allow: /blogs/pad/$ for *
  • Removed Allow: /people/$ for *
  • Removed Allow: /people/pad$ for *
  • … 45 more change(s)
view diff → snapshot
Eli Lilly lilly.com modified +2 −0
  • Added Disallow: /uk/lillydirect/ for *
  • Added Disallow: /uk/lillydirect/get-started for *
view diff → snapshot
Dell Technologies dell.com modified +1 −8
  • Removed Disallow: */lp/ for *
  • Removed Disallow: */shop/*/scr/* for *
  • Removed Disallow: */shop/dell-parts/ar/*/platform=* for *
  • Removed Disallow: */shop/dell-parts/partsbytype/* for *
  • Removed Disallow: */shop/partsbytype/* for *
  • Removed Disallow: */shop/pfydresults/* for *
  • … 1 more change(s)
view diff → snapshot
Figma Community figma.com modified +9 −0
  • 🤖 AI Added Allow: /resource-library/ai-art-styles/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-product-photography-examples/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/ai-storyboard-examples/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/contextual-ai-coding/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-system-governance/$ for CHATGPT-User
  • 🤖 AI Added Allow: /resource-library/design-token-architecture/$ for CHATGPT-User
  • … 39 more change(s)
view diff → snapshot
Fox News foxnews.com modified +3 −1
  • Added Disallow: /api/article-search for *
  • Added Disallow: /xid for *
  • Removed sitemap https://video.foxnews.com/videositemap.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Baxter International baxter.com modified +2 −133
  • 🤖 AI Unblocked Amazonbot (Amazon) - rules removed
  • 🤖 AI Unblocked Applebot-Extended (Apple) - rules removed
  • 🤖 AI Unblocked Bytespider (ByteDance) - rules removed
  • 🤖 AI Unblocked CCBot (Common Crawl) - rules removed
  • 🤖 AI Unblocked ClaudeBot (Anthropic) - rules removed
  • Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
  • … 45 more change(s)
view diff → snapshot
Dell Technologies dell.com modified +11 −5
  • Added Disallow: */lp/ for *
  • Added Disallow: */sac/*/*/*/appref=* for *
  • Added Disallow: */shop/*/scr/* for *
  • Added Disallow: */shop/dell-parts/ar/*/platform=* for *
  • Added Disallow: */shop/dell-parts/partsbytype/* for *
  • Added Disallow: */shop/partsbytype/* for *
  • … 6 more change(s)
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for Claude-User
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /ja-jp/blog/japan-local-data-hosting/$ for PerplexityBot
view diff → snapshot