NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1198files archived
2614snapshots
1442changes detected
231this week
All changes 🤖 AI-crawler changes Showing initial archives
Goodyear Tire & Rubber goodyear.com modified +0 −40
  • Removed all rules for user-agent ‘ahrefsbot’
  • Removed all rules for user-agent ‘cazoodlebot’
  • Removed all rules for user-agent ‘dotbot/1.0’
  • Removed all rules for user-agent ‘gigabot’
  • Removed all rules for user-agent ‘mj12bot’
  • Removed all rules for user-agent ‘sogou inst spider’
  • … 20 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Figma Community figma.com modified +1 −0
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for CHATGPT-User
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for OAI-SearchBot
  • 🤖 AI Added Allow: /customers/mitsubishi-electric-corporation/$ for PerplexityBot
view diff → snapshot
Snapchat snapchat.com modified +3 −3
  • Removed Disallow: /local/ for *
  • Removed Disallow: /local/ for Googlebot
  • Removed Disallow: /local/ for Meta-WebIndexer
  • Added sitemap https://www.snapchat.com/sitemaps/local_city_hub/sitemap-index.xml
  • Added sitemap https://www.snapchat.com/sitemaps/local_city_leaf/sitemap-index.xml
  • Added sitemap https://www.snapchat.com/sitemaps/local_nbhd_leaf/sitemap-index.xml
view diff → snapshot
Eastman Chemical eastman.com modified +2 −0
  • Added Disallow: /Private/ for *
  • Added Disallow: /content/dam/eastman/corporate/en/literature/private/ for *
view diff → snapshot
Avery Dennison averydennison.com modified +1 −1
  • Added sitemap https://www.averydennison.com/sitemap-main.xml
  • Removed sitemap https://www.averydennison.com/en/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
DSV dsv.com created +34 −32
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-bg-bg-sitemap_index.xml
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-cs-cz-sitemap_index.xml
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-da-dk-sitemap_index.xml
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-de-de-sitemap_index.xml
  • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-ae-sitemap_index.xml
  • … 45 more change(s)
view diff → snapshot
South China Morning Post scmp.com modified +3 −0
  • Added Disallow: /native/business/banking-finance/topics/wealth-across-borders/article/3357488/retired-businessman-and-lifelong-sailor-trust-anchors-process-building-wealth for Googlebot
  • Added Disallow: /print/native/business/banking-finance/topics/wealth-across-borders/article/3357488/retired-businessman-and-lifelong-sailor-trust-anchors-process-building-wealth for Googlebot
  • Added Disallow: /print/presented/better-life/career-education/topics/edtalk-harbour-school/article/3199785/rethinking-5-educational-myths-through-experiential-learning-approach for Googlebot
view diff → snapshot
Canva canva.com modified +2 −0
  • Added Disallow: /*/help/*&query= for *
  • Added Disallow: /*/help/*?query= for *
view diff → snapshot
Barracuda barracuda.com created +28 −28
  • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for *
  • Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for *
  • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for DotBot
  • Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for DotBot
  • Added Disallow: /tUBYNifRs1e8unWE00sUsxLRZN11tkvP-DXPflxM6aY=.html for sistrix
  • Removed Disallow: /Zy_-gwOInfBYquGfBHrXiYArtDznhkwu8FzslHaGrfI=.html for sistrix
  • … 2 more change(s)
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Barracuda barracuda.com removed +0 −232
  • robots.txt is no longer served (HTTP 404)
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
Lowe's lowes.com modified +128 −19
  • Added Disallow: /pd/*/af$ for *
  • Added Disallow: /pd/*/af=$ for *
  • Added Disallow: /pd/*/ar$ for *
  • Added Disallow: /pd/*/az$ for *
  • Added Disallow: /pd/*/be$ for *
  • Added Disallow: /pd/*/bg$ for *
  • … 45 more change(s)
view diff → snapshot