NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5436snapshots
4301changes detected
236this week
All changes 🤖 AI-crawler changes Hiding initial archives
Nubank nubank.com.br modified +1392 −0
  • Added Disallow: /ajuda-e-seguranca for *
  • Added Disallow: /ajuda-e-seguranca/central-de-protecao for *
  • Added Disallow: /ajuda-e-seguranca/rotativo-e-parcelamento for *
  • Added Disallow: /ajuda-e-seguranca/selo-de-prevencao-a-fraude for *
  • Added Disallow: /buscar for *
  • Added Disallow: /chatgpt-go for *
  • … 45 more change(s)
view diff → snapshot
MetLife metlife.com modified +2 −8
  • Added Disallow: /investments$ for *
  • Added Disallow: /investments/ for *
  • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Figma Community figma.com modified +17 −0
  • 🤖 AI Added Allow: /blog/properties-panel-and-annotations-now-in-figma-make/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-annotation-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-background-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-card-layout-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-color-palette-generator/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-empty-state-generator/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-us-eb-buyers-guide-choosing-dspm-solution-ai-era.pdf for *
view diff → snapshot
Walgreens Boots Alliance walgreens.com modified +4 −0
  • Added Disallow: /buyagain/product/ for *
  • Added Disallow: /digitalretailbff/v1/sdui/ for *
  • Added Disallow: /productrectransformer/v1/ for *
  • Added Disallow: /retailsearch/ for *
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://127.0.0.1/image-sitemap.xml
  • Added sitemap http://127.0.0.1/page-sitemap.xml
  • Removed sitemap http://www.iqvia.com/image-sitemap.xml
  • Removed sitemap http://www.iqvia.com/page-sitemap.xml
view diff → snapshot
Claude claude.ai modified +1 −1
  • 🤖 AI Blocked Google-Extended (Google) entirely
  • Removed Disallow: /share/* for *
view diff → snapshot
Figma Community figma.com modified +3 −0
  • 🤖 AI Added Allow: /solutions/ai-frame-namer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-page-organizer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-variable-namer/$ for CHATGPT-User
  • 🤖 AI Added Allow: /solutions/ai-frame-namer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /solutions/ai-page-organizer/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /solutions/ai-variable-namer/$ for Claude-SearchBot
  • … 6 more change(s)
view diff → snapshot
Walmart walmart.com modified +1 −1
  • Added sitemap https://www.walmart.com/sitemap_product_03.xml
  • Removed sitemap https://www.walmart.com/sitemap_itp_03.xml
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
IQVIA Holdings iqvia.com modified +2 −2
  • Added sitemap http://www.iqvia.com/image-sitemap.xml
  • Added sitemap http://www.iqvia.com/page-sitemap.xml
  • Removed sitemap http://127.0.0.1/image-sitemap.xml
  • Removed sitemap http://127.0.0.1/page-sitemap.xml
view diff → snapshot
MetLife metlife.com modified +8 −2
  • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
  • Added Disallow: https://metlife-prod.adobecqms.net/* for *
  • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
  • Added Disallow: https://metlife-qa1-65.adobecqms.net/* for *
  • … 3 more change(s)
view diff → snapshot
Bild bild.de modified +4 −0
  • Added Disallow: /buehne/ for *
  • Added Disallow: /sonstiges/intern/ for *
  • Added Disallow: /vorprod/ for *
  • Added Disallow: /vorproduktion/ for *
view diff → snapshot
University College London ucl.ac.uk modified +471 −484
  • Added sitemap https://www.ucl.ac.uk/bloomsbury-theatre/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/centre-law-europe/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/cms-archive/2034/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/cms-archive/3dgeoinfo/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/cms-archive/about/sitemap.xml
  • Added sitemap https://www.ucl.ac.uk/cms-archive/academic-manual/sitemap.xml
  • … 45 more change(s)
view diff → snapshot
Linde linde.com modified +2 −1
  • Unblocked user-agent ‘*’
  • Added Disallow: /search for *
  • Added sitemap https://www.linde.com/sitemap.xml
view diff → snapshot