NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1204files archived
3937snapshots
2782changes detected
216this week
All changes 🤖 AI-crawler changes Hiding initial archives
Pika pika.art modified +3 −0
  • Added Disallow: /credits for *
  • Added Disallow: /jobs for *
  • Added Disallow: /subscription for *
view diff → snapshot
Proofpoint proofpoint.com modified +6 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-au-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-de-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-es-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-fr-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-it-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-uk-eb-email-security-blind-spots.pdf for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Suno suno.com modified +2 −0
  • Added sitemap https://suno.com/products-sitemap.xml
  • Added sitemap https://suno.com/tools-sitemap.xml
view diff → snapshot
GoDaddy godaddy.com modified +32 −11
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Added Allow: */domain-auctions/ for Claude-SearchBot
  • 🤖 AI Added Allow: */domainsearch/ for Claude-SearchBot
  • 🤖 AI Added Disallow: */domain-auctions/api/* for Claude-SearchBot
  • 🤖 AI Removed Allow: /domain-auctions/ for Claude-SearchBot
  • … 26 more change(s)
view diff → snapshot
Corteva corteva.com modified +2 −1
  • Added Disallow: /content/dam/dpagco/corteva/global/corporate/files/code-of-conduct/Corteva_Code_Interactive_ruRU_Russian.pdf for *
  • Added Disallow: /content/dam/dpagco/corteva/global/corporate/general/files/code-of-conduct/Corteva%20QRG%20Russian.pdf for *
  • Removed Disallow: /content/dam/dpagco/ for *
view diff → snapshot
Figma Community figma.com modified +14 −6
  • 🤖 AI Added Allow: /blog/show-your-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Slate slate.com modified +2 −1
  • Added rules for user-agent ‘Dataprovider’
  • Unblocked user-agent ‘Dataprovider.com’
  • Added Disallow: /_ for Dataprovider.com
  • Added Disallow: /comments/ for Dataprovider.com
  • Added Disallow: /css for Dataprovider.com
  • Added Disallow: /fonts for Dataprovider.com
  • … 5 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +5 −1
  • Unblocked user-agent ‘Applebot’
  • Added Allow: / for Applebot
  • 🤖 AI Added Allow: /sponsored/ for Claude-SearchBot
  • 🤖 AI Added Allow: /sponsored/ for Claude-User
  • 🤖 AI Added Allow: /sponsored/ for Perplexity-User
  • 🤖 AI Added Allow: /sponsored/ for PerplexityBot
view diff → snapshot
Penguin Random House penguinrandomhouse.com modified +1 −10
  • Blocked user-agent ‘*’ entirely
  • Removed Allow: /wp-admin/admin-ajax.php for *
  • Removed Allow: /wp-includes/css/dist/block-library/style.min.css for *
  • Removed Disallow: /interactive/reading-preference for *
  • Removed Disallow: /prh-internal-news/ for *
  • Removed Disallow: /search/ for *
  • … 5 more change(s)
view diff → snapshot
Best Buy bestbuy.com modified +0 −11
  • Removed Disallow: */site/searchpage.jsp?st=* for *
  • Removed Disallow: *?cp=*&id=*&st=* for *
  • Removed Disallow: *?id*_facet*&st=* for *
  • Removed Disallow: *?id*_facet*_facet*&st=* for *
  • 🤖 AI Removed Disallow: *?cp=*&id=*&st=* for AmazonBot
  • 🤖 AI Removed Disallow: *?id*_facet*&st=* for AmazonBot
  • … 45 more change(s)
view diff → snapshot