- Added Disallow: /markets/energy/crude-oil/commodity-direct-spot-crude-markets.html for *
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1198files archived
2643snapshots
1471changes detected
226this week
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /boas-vindas-a-cconta-nu-empresas for *
- Added Disallow: /boas-vindas-a-conta-nu-empresas for *
- Added Disallow: /boas-vindas-a-cconta-nu-empresas for DatoCmsSearchBot
- Added Disallow: /boas-vindas-a-conta-nu-empresas for DatoCmsSearchBot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /*&fbclid= for *
- Added Disallow: /*&gb= for *
- Added Disallow: /*&playlist= for *
- Added Disallow: /*&q= for *
- Added Disallow: /*&traffic_source= for *
- Added Disallow: /*?fbclid= for *
- … 4 more change(s)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for CHATGPT-User
- 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for Claude-SearchBot
- 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for OAI-SearchBot
- 🤖 AI Added Allow: /es-la/customers/how-insider-one-customers-convert-designs-into-campaigns-instantly-with-figma-mcp/$ for PerplexityBot
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /component-definition.json for *
- Added Disallow: /component-filters.json for *
- Added Disallow: /component-models.json for *
- Added Disallow: /content/pearson-sites/language-masters/* for *
- Added Disallow: /en-us/fragments/* for *
- Added Disallow: /en-us/platinum-test/* for *
- … 1 more change(s)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added Disallow: /presented/better-life/career-education/topics/edtalk-harbour-school/article/3199785/rethinking-5-educational-myths-through-experiential-learning-approach for Googlebot
- Added Allow: /core/*.avif for *
- Added Allow: /core/*.webp for *
- Added Allow: /profiles/*.avif for *
- Added Allow: /profiles/*.webp for *
- Added Disallow: /index.php/search? for *
- Added Disallow: /search? for *
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- Formatting/comment-only change (no semantic difference)
- First snapshot of robots.txt archived
- Added sitemap http://127.0.0.1/image-sitemap.xml
- Added sitemap http://127.0.0.1/page-sitemap.xml
- Removed sitemap http://www.iqvia.com/image-sitemap.xml
- Removed sitemap http://www.iqvia.com/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)
- Added sitemap http://www.iqvia.com/image-sitemap.xml
- Added sitemap http://www.iqvia.com/page-sitemap.xml
- Removed sitemap http://127.0.0.1/image-sitemap.xml
- Removed sitemap http://127.0.0.1/page-sitemap.xml
- Formatting/comment-only change (no semantic difference)