NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5416snapshots
4280changes detected
237this week
All changes 🤖 AI-crawler changes Hiding initial archives
Figma Community figma.com modified +8 −0
  • 🤖 AI Added Allow: /blog/workflow-lab-build-with-code-on-the-canvas-using-the-figma-agent-and-code-layers/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /fr-fr/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • 🤖 AI Added Allow: /ja-jp/blog/source-material-brent-david-freaney/$ for CHATGPT-User
  • … 34 more change(s)
view diff → snapshot
Varonis varonis.com modified +1 −1
  • Added sitemap https://varonis-mcp.webflow.io/sitemap.xml
  • Removed sitemap https://webflow-www.varonis.com/sitemap.xml
view diff → snapshot
Nubank nubank.com.br modified +0 −18
  • Removed Disallow: /home for *
  • Removed Disallow: /home2 for *
  • Removed Disallow: /home3 for *
  • Removed Disallow: /home for DatoCmsSearchBot
  • Removed Disallow: /home2 for DatoCmsSearchBot
  • Removed Disallow: /home3 for DatoCmsSearchBot
view diff → snapshot
StoneX Group stonex.com modified +9 −9
  • Added Disallow: /*/bin/ for *
  • Added Disallow: /*/error/ for *
  • Added Disallow: /*/lp/ for *
  • Added Disallow: /*/search/?query= for *
  • Added Disallow: /*/sitecore modules for *
  • Added Disallow: /*/sitecore/ for *
  • … 12 more change(s)
view diff → snapshot
Figma Community figma.com modified +2 −0
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for CHATGPT-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for CHATGPT-User
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for Claude-SearchBot
  • 🤖 AI Added Allow: /blog/how-figma-manages-sandboxes-at-scale/$ for Claude-User
  • 🤖 AI Added Allow: /legal/digital-regulation-help-centre/brazil-general-data-protection-law/$ for Claude-User
  • … 4 more change(s)
view diff → snapshot
Pinterest pinterest.com modified +103 −101
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261212_day_of_the_virgin_of_guadalupe_www.pinterest.com.xml
  • Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261212_feast_of_our_lady_of_guadalupe_www.pinterest.com.xml
view diff → snapshot
SSRN ssrn.com removed +0 −14
  • robots.txt is no longer served (HTTP 404)
Threads threads.net modified +1 −0
  • Added sitemap https://www.threads.com/sitemap/threads_tech_posts_sitemap.xml.gz
view diff → snapshot
Glencore glencore.com modified +3 −3
  • Added Disallow: /.rest/api/v1/documents/static/610492b3-3d6e-4622-aff1-4fc0f747b271/Business+confirmation+general+terms+for+purchases+-+basic.pdf for *
  • Added Disallow: /.rest/api/v1/documents/static/72f40bbd-1240-4e41-8177-5fba47c6323f/Business+confirmation+general+terms+for+purchases+-+extended.pdf for *
  • Added Disallow: /.rest/api/v1/documents/static/975c896c-6824-401f-9def-50bc0d31d097/Business+confirmation+general+terms+for+sales.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/610492b3-3d6e-4622-aff1-4fc0f747b271/Business%2Bconfirmation%2Bgeneral%2Bterms-purchase.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/72f40bbd-1240-4e41-8177-5fba47c6323f/Business%2Bconfirmation%2Bgeneral%2Bterms-purchase-ext.pdf for *
  • Removed Disallow: /.rest/api/v1/documents/static/975c896c-6824-401f-9def-50bc0d31d097/Business%2Bconfirmation%2Bgeneral%2Bterms-sale.pdf for *
view diff → snapshot
Lam Research lamresearch.com created +7 −6
  • Added sitemap https://www.lamresearch.com/de/sitemap.xml
  • Added sitemap https://www.lamresearch.com/ja/sitemap.xml
  • Added sitemap https://www.lamresearch.com/ko/sitemap.xml
  • Added sitemap https://www.lamresearch.com/sitemap.xml
  • Added sitemap https://www.lamresearch.com/zh-hans/sitemap.xml
  • Added sitemap https://www.lamresearch.com/zh-hant/sitemap.xml
  • … 1 more change(s)
view diff → snapshot
ACS Group grupoacs.com modified +2 −2
  • Added Disallow: /accionistas-e-inversores/eventos-resultados-y-presentaciones/eventos-corporativos-y-presentaciones/capital-markets-day-2026 for *
  • Added Disallow: /en/shareholders-investors/events-results-and-presentations/corporate-events-and-presentations/capital-markets-day-2026 for *
  • Removed Disallow: /accionistas-e-inversores-nuevo/* for *
  • Removed Disallow: /shareholders-investors-new/* for *
view diff → snapshot
Corriere della Sera corriere.it modified +160 −2
  • 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
  • Blocked user-agent ‘Ai2Bot-Dolma’ entirely
  • Blocked user-agent ‘Amazon Kendra’ entirely
  • Blocked user-agent ‘Arquivo-web-crawler’ entirely
  • Blocked user-agent ‘Barkrowler’ entirely
  • Blocked user-agent ‘Big Sur AI’ entirely
  • … 45 more change(s)
view diff → snapshot
Shell shell.com modified +0 −2
  • Removed Disallow: /business-customers/marine/about-shell-marine.html for *
  • Removed Disallow: /business-customers/marine/about-shell-marine/ for *
view diff → snapshot
Molina Healthcare molinahealthcare.com modified +28 −3
  • Added Disallow: /-/media/Molina/PublicWebsite/PDF/ for *
  • Added Disallow: /marketplace/az/* for *
  • Added Disallow: /marketplace/ca/* for *
  • Added Disallow: /marketplace/fl/* for *
  • Added Disallow: /marketplace/ga/* for *
  • Added Disallow: /marketplace/ia/* for *
  • … 24 more change(s)
view diff → snapshot
Samsung Electronics samsung.com modified +2 −1
  • Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for *
  • Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for *
  • Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for AdsBot-Google
  • Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for AdsBot-Google
  • 🤖 AI Added Allow: /sg/multistore/sg_eppfinancing/offers_epp/ for ChatGPT-User
  • 🤖 AI Removed Allow: /sg/multistore/sg_eppfinancing/offers_epp/sc_premium/ for ChatGPT-User
  • … 12 more change(s)
view diff → snapshot