The Pulse

Every change to a monitored robots.txt, archived and explained.

1273sites tracked

1206files archived

5428snapshots

4292changes detected

249this week

All changes 🤖 AI-crawler changes Hiding initial archives

Adidas adidas.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Southwest Airlines southwest.com modified +1 −0 1mo ago

Added sitemap https://support.southwest.com/helpcenter/sfsites/c/resource/HelpCenter_Sitemap

view diff → snapshot

DSV dsv.com modified +24 −24 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Tencent tencent.com created +0 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Sysco sysco.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Imperial College London imperial.ac.uk modified +1 −0 1mo ago

Added Disallow: /study/courses/postgraduate-taught/2026/ for *

view diff → snapshot

Pinterest pinterest.com modified +52 −47 1mo ago

Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261130_day_of_remembrance_for_all_victims_of_chemical_war_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261203_international_day_of_persons_with_disabilities_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_hanukkah_day_1_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_st_nicholas_eve_sinterklaas_www.pinterest.com.xml
Added sitemap https://www.pinterest.com/v3_sitemaps/seasonal_events_sitemap_20261205_world_soil_day_www.pinterest.com.xml

view diff → snapshot

CNN cnn.com modified +2 −4 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Shell shell.com modified +2 −0 1mo ago

Added Disallow: /shellenergy/cement.html for *
Added Disallow: /shellenergy/cement/ for *

view diff → snapshot

Colgate-Palmolive colgatepalmolive.com modified +11 −2 1mo ago

Added rules for user-agent ‘ColPal-Elastic-Crawler’
Added rules for user-agent ‘Elastic-Crawler’
Added rules for user-agent ‘Swiftbot’
Added sitemap https://www.colgatepalmolive.com/en-us/sitemap.xml
Added sitemap https://www.colgatepalmolive.com/et-ee/sitemap.xml
Removed sitemap /sitemap.xml

view diff → snapshot

Southwest Airlines southwest.com modified +0 −1 1mo ago

Removed sitemap https://support.southwest.com/helpcenter/sfsites/c/resource/HelpCenter_Sitemap

view diff → snapshot

DSV dsv.com modified +23 −23 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Intesa Sanpaolo intesasanpaolo.com modified +0 −1 1mo ago

Removed Disallow: /content/dam/vetrina/documenti/landing/anti-phishing/ for *

view diff → snapshot

WordPress wordpress.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Tencent tencent.com removed +0 −2 1mo ago

robots.txt is no longer served (HTTP 404)

Toronto Star thestar.com modified +14 −14 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Schneider Electric se.com modified +6 −1 1mo ago

Added sitemap https://www.se.com/bg/bg/faqs/sitemap.xml
Added sitemap https://www.se.com/ca/fr/faqs/sitemap.xml
Added sitemap https://www.se.com/rs/sr/faqs/sitemap.xml

view diff → snapshot

Wells Fargo wellsfargo.com modified +2 −0 1mo ago

Added Disallow: /biz/business-credit/agreements/ for *
Added Disallow: /biz/business-credit/credit-cards/ for *

view diff → snapshot

Cohesity cohesity.com modified +10 −2 1mo ago

Removed Disallow: /forms/free-trial/dmaas-stage/ for *
Added sitemap https://www.cohesity.com/sitemap.br.xml
Added sitemap https://www.cohesity.com/sitemap.de.xml
Added sitemap https://www.cohesity.com/sitemap.es.xml
Added sitemap https://www.cohesity.com/sitemap.fr.xml
Added sitemap https://www.cohesity.com/sitemap.jp.xml
… 3 more change(s)

view diff → snapshot

Proofpoint proofpoint.com modified +1 −0 1mo ago

Added Disallow: /sites/default/files/misc/202609_wbr_jp_ZenGuide-Pathway.pdf for *

view diff → snapshot

KLA kla.com modified +1 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Bank Central Asia bca.co.id modified +1 −1 1mo ago

Added sitemap http://www.bca.co.id/sitemap.xml
Removed sitemap https://www.bca.co.id/sitemap.xml

view diff → snapshot

WordPress wordpress.com modified +1 −1 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Proofpoint proofpoint.com modified +1 −0 1mo ago

Added Disallow: /sites/default/files/e-books/pfpt-kr-eb-stopping-hijacked-business-communications_0.pdf for *

view diff → snapshot

DSV dsv.com modified +19 −19 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Teva Pharmaceutical teva.com modified +1 −0 1mo ago

Added sitemap https://teva.com/sitemap-llms.xml

view diff → snapshot

Amgen amgen.com modified +2 −0 1mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot