- Added Disallow: /reject-all for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5428snapshots
4292changes detected
227this week
- Added Disallow: /reject-all for *
- Added Disallow: /sites/default/files/white-papers/pfpt-wp-us-votc-2026.pdf for *
- Removed Disallow: /en_us/careers/search-jobs.html? for *
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘AhrefsBot’
- Removed all rules for user-agent ‘DDM-DCipher’
- 🤖 AI Removed rules for AI crawler GPTBot (OpenAI)
- Removed all rules for user-agent ‘rogerbot’
- Blocked user-agent ‘*’ entirely
- Removed Allow: /learn/ for *
- … 31 more change(s)
- Unblocked user-agent ‘mediapartners-google’
- Removed Allow: /watchlist for mediapartners-google
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Blocked user-agent ‘archive.org_bot’ entirely
- Blocked user-agent ‘heritrix’ entirely
- Blocked user-agent ‘ia_archiver’ entirely
- Blocked user-agent ‘ia_archiver-web.archive.org’ entirely
- Blocked user-agent ‘ia_archiver2’ entirely
- Blocked user-agent ‘Special_Archiver’ entirely
- … 1 more change(s)
- 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for CHATGPT-User
- 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for Claude-SearchBot
- 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for Claude-User
- 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for OAI-SearchBot
- 🤖 AI Added Allow: /pl-pl/rozwiazania/generator-systemow-projektowania-ai/$ for PerplexityBot
- robots.txt is no longer served (HTTP 404)
- Blocked user-agent ‘Bravebot’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- Blocked user-agent ‘factset_spyderbot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘Shap-User’ entirely
- … 3 more change(s)
- Blocked user-agent ‘Bravebot’ entirely
- Blocked user-agent ‘Brightbot 1.0’ entirely
- Blocked user-agent ‘factset_spyderbot’ entirely
- Blocked user-agent ‘FirecrawlAgent’ entirely
- Blocked user-agent ‘Scrapy’ entirely
- Blocked user-agent ‘Shap-User’ entirely
- … 3 more change(s)
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/future-solutions-factory.html for *
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/future-solutions-factory/ for *
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/how-biodegradable-lubricants-can-support-sustainable-construction.html for *
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/how-biodegradable-lubricants-can-support-sustainable-construction/ for *
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/leadership-in-manufacturing.html for *
- Added Disallow: /business-customers/industrial-lubricants-and-specialty-fluids-for-business/perspectives/leadership-in-manufacturing/ for *
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘Baiduspider’
- Removed all rules for user-agent ‘BingPreview’
- 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Removed rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Removed rules for AI crawler Google-Extended (Google)
- Removed all rules for user-agent ‘Googlebot’
- … 22 more change(s)
- Formatting/comment-only change (no semantic difference)
- Added rules for user-agent ‘*’
- Added rules for user-agent ‘Mediapartners-Google’
- Added sitemap https://shop.advanceautoparts.com/sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AI Added rules for AI crawler ChatGPT-User (OpenAI)
- 🤖 AI Added rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AI Added rules for AI crawler Google-Extended (Google)
- 🤖 AI Added rules for AI crawler GPTBot (OpenAI)
- 🤖 AI Added rules for AI crawler PerplexityBot (Perplexity)
- … 2 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘baiduspider’
- Removed all rules for user-agent ‘bingbot’
- 🤖 AI Removed rules for AI crawler ChatGPT-User (OpenAI)
- Removed all rules for user-agent ‘Googlebot’
- Removed all rules for user-agent ‘MSNbot’
- 🤖 AI Removed rules for AI crawler OAI-SearchBot (OpenAI)
- … 19 more change(s)
- Formatting/comment-only change (no semantic difference)
- Removed all rules for user-agent ‘*’
- Removed all rules for user-agent ‘Mediapartners-Google’
- Removed sitemap https://shop.advanceautoparts.com/sitemap.xml