NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +21 −0
  • 🤖 AI Added Allow: /de-de/blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/build-by-design/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-manon-macasaet/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/websites-with-payload-in-figma/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Roblox roblox.com modified +2 −0
  • Added sitemap https://www.roblox.com/sitemap-games-nl.xml
  • Added sitemap https://www.roblox.com/sitemap-nl.xml
view diff → snapshot
Proofpoint proofpoint.com modified +1 −0
  • Added Disallow: /sites/default/files/threat-reports/pfpt-jp-rt-2026-ai-era-ransomware_v2.pdf for *
view diff → snapshot
StoneX Group stonex.com modified +2 −7
  • Removed Disallow: /*~/ for *
  • Added sitemap https://www.stonex.com/es-ar/sitemap-index.xml
  • Added sitemap https://www.stonex.com/ja-jp/sitemap-index.xml
  • Removed sitemap https://www.stonex.com/en-gb/sitemap-b2b.xml
  • Removed sitemap https://www.stonex.com/en-gb/sitemap-b2c.xml
  • Removed sitemap https://www.stonex.com/en/sitemap-b2b.xml
  • … 3 more change(s)
view diff → snapshot
SSRN ssrn.com removed +0 −14
  • robots.txt is no longer served (HTTP 404)
Instagram instagram.com modified +1 −0
  • Added sitemap https://www.instagram.com/sitemap/ig_seo_profile_sitemap_v2.xml.gz
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +3 −28
  • Added Disallow: (empty) for *
  • Removed Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Removed Disallow: /marketplace/az/* for *
  • Removed Disallow: /marketplace/ca/* for *
  • Removed Disallow: /marketplace/fl/* for *
  • Removed Disallow: /marketplace/ga/* for *
  • … 24 more change(s)
view diff → snapshot
University of Cape Town uct.ac.za modified +1 −1
  • Added sitemap https://uct.ac.za/sitemap.xml
  • Removed sitemap https://uct.ac.za/sitemap.xmlThe website encountered an unexpected error. Try again later.<br />
view diff → snapshot
Max max.com removed +0 −3
  • robots.txt is no longer served (HTTP 404)