NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Showing initial archives
Toronto Star thestar.com modified +21 −19
  • Added Disallow: /subscribe/trial-purchase-success/ for *
  • Added Disallow: /subscribe/trial-purchase/ for *
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +67 −0
  • 🤖 AI Added Allow: /customers/how-carrefour-leveraged-ai-to-drive-design-delivery/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/in-good-company-how-publishers-use-figma-to-help-design-the-news/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/7-questions-from-lc/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/design-systems-104-making-metrics-matter/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
Schneider Electric se.com modified +8 −0
  • Added sitemap https://www.schneider-electric.cn/zh/download/sitemapindex/
  • Added sitemap https://www.schneider-electric.cn/zh/faqs/sitemap.xml
  • Added sitemap https://www.schneider-electric.cn/zh/product-category/google-category-sitemapindex-CN-zh.xml
  • Added sitemap https://www.schneider-electric.cn/zh/product-range/google-range-sitemapindex-CN-zh.xml
  • Added sitemap https://www.schneider-electric.cn/zh/product-subcategory/google-subcategory-sitemapindex-CN-zh.xml
  • Added sitemap https://www.schneider-electric.cn/zh/product/google-product-sitemapindex-CN-zh.xml
  • … 1 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Heidelberg Materials heidelbergmaterials.com modified +1 −1
  • Added sitemap https://www.heidelbergmaterials.com/index/sitemap.xml
  • Removed sitemap https://www.heidelbergmaterials.com/sitemap.xml
view diff → snapshot
Realtor realtor.com modified +4 −0
  • Added Allow: /realestateandhomes-search/*/price-na- for *
  • Added Disallow: /realestateandhomes-search/*/price- for *
  • Added Disallow: /realestateandhomes-search/*/price-na-*/pg- for *
view diff → snapshot
Nubank nubank.com.br modified +6 −0
  • Added Disallow: /nu-design-day-inscricao for *
  • Added Disallow: /nu-design-day-inscricao for DatoCmsSearchBot
view diff → snapshot