NeuralCrawl

The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked
1206files archived
5428snapshots
4292changes detected
249this week
All changes 🤖 AI-crawler changes Hiding initial archives
Indeed indeed.com modified +33 −17
  • Removed all rules for user-agent ‘Bingbot’
  • Added Disallow: /Zeige-Job for *
  • Added Disallow: /generate-question for *
  • Added Disallow: /hire/ats-partners for *
  • Added Disallow: /hire/o/ats-partners for *
  • Added Disallow: /m/viewjob for *
  • … 45 more change(s)
view diff → snapshot
Fortra fortra.com modified +7 −0
  • Added rules for user-agent ‘SemrushBot’
  • Added rules for user-agent ‘SiteAuditBot’
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +27 −0
  • Added rules for user-agent ‘*’
  • Added rules for user-agent ‘Mediapartners-Google’
  • Added sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Commonwealth Bank commbank.com.au modified +4 −1
  • Added Allow: /etc/designs/commbank/favicon.ico for *
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Amazonbot
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Amazonbot
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Applebot-Extended
  • 🤖 AI Added Allow: /etc/designs/commbank/favicon.ico for Applebot-Extended
  • 🤖 AI Added Allow: /digital/home-loans/borrowing-power-calculator for Bytespider
  • … 29 more change(s)
view diff → snapshot
Pika pika.art modified +3 −0
  • Added Disallow: /credits for *
  • Added Disallow: /jobs for *
  • Added Disallow: /subscription for *
view diff → snapshot
Proofpoint proofpoint.com modified +6 −0
  • Added Disallow: /sites/default/files/e-books/pfpt-au-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-de-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-es-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-fr-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-it-eb-email-security-blind-spots.pdf for *
  • Added Disallow: /sites/default/files/e-books/pfpt-uk-eb-email-security-blind-spots.pdf for *
view diff → snapshot
Advance Auto Parts advanceautoparts.com modified +0 −27
  • Removed all rules for user-agent ‘*’
  • Removed all rules for user-agent ‘Mediapartners-Google’
  • Removed sitemap https://shop.advanceautoparts.com/sitemap.xml
view diff → snapshot
Suno suno.com modified +2 −0
  • Added sitemap https://suno.com/products-sitemap.xml
  • Added sitemap https://suno.com/tools-sitemap.xml
view diff → snapshot
GoDaddy godaddy.com modified +32 −11
  • 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
  • 🤖 AI Added rules for AI crawler Perplexity-User (Perplexity)
  • 🤖 AI Added Allow: */domain-auctions/ for Claude-SearchBot
  • 🤖 AI Added Allow: */domainsearch/ for Claude-SearchBot
  • 🤖 AI Added Disallow: */domain-auctions/api/* for Claude-SearchBot
  • 🤖 AI Removed Allow: /domain-auctions/ for Claude-SearchBot
  • … 26 more change(s)
view diff → snapshot
Corteva corteva.com modified +2 −1
  • Added Disallow: /content/dam/dpagco/corteva/global/corporate/files/code-of-conduct/Corteva_Code_Interactive_ruRU_Russian.pdf for *
  • Added Disallow: /content/dam/dpagco/corteva/global/corporate/general/files/code-of-conduct/Corteva%20QRG%20Russian.pdf for *
  • Removed Disallow: /content/dam/dpagco/ for *
view diff → snapshot
Figma Community figma.com modified +14 −6
  • 🤖 AI Added Allow: /blog/show-your-work/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /de-de/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/source-material-thom-bettridge/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-es/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • 🤖 AI Added Allow: /es-la/blog/workflow-lab-moving-between-design-and-code-with-agents/$ for CHATGPT-User
  • … 45 more change(s)
view diff → snapshot
Slate slate.com modified +2 −1
  • Added rules for user-agent ‘Dataprovider’
  • Unblocked user-agent ‘Dataprovider.com’
  • Added Disallow: /_ for Dataprovider.com
  • Added Disallow: /comments/ for Dataprovider.com
  • Added Disallow: /css for Dataprovider.com
  • Added Disallow: /fonts for Dataprovider.com
  • … 5 more change(s)
view diff → snapshot
The Atlantic theatlantic.com modified +5 −1
  • Unblocked user-agent ‘Applebot’
  • Added Allow: / for Applebot
  • 🤖 AI Added Allow: /sponsored/ for Claude-SearchBot
  • 🤖 AI Added Allow: /sponsored/ for Claude-User
  • 🤖 AI Added Allow: /sponsored/ for Perplexity-User
  • 🤖 AI Added Allow: /sponsored/ for PerplexityBot
view diff → snapshot