NeuralCrawl — change feed

DSV dsv.com modified +27 −27 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Amgen amgen.com modified +0 −2 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Toronto Star thestar.com modified +14 −14 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Barracuda barracuda.com created +27 −27 2mo ago

Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for *
Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for *
Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for DotBot
Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for DotBot
Added Disallow: /GivfqzeDvfdw9hMC07duOnQLNMB0MS0XyvQGw5MZSb8=.html for sistrix
Removed Disallow: /NtNR1a6g05Bg2Qe828G_L4xM52ld7d3h5NtuUhyr9Dg=.html for sistrix
… 2 more change(s)

view diff → snapshot

Vimeo vimeo.com modified +0 −1 2mo ago

Removed Disallow: /showcase/*/auth for *

view diff → snapshot

Ludwig Maximilian University of Munich lmu.de modified +2 −0 2mo ago

Added Disallow: /center-geoastronomy/site_tech for *
Added sitemap https://www.lmu.de/center-geoastronomy/sitemap.xml

view diff → snapshot

Snapchat snapchat.com modified +0 −5 2mo ago

Removed sitemap https://www.snapchat.com/sitemaps/curated_topic/sitemap-index.xml
Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_all/sitemap-index.xml
Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_eight_plus/sitemap-index.xml
Removed sitemap https://www.snapchat.com/sitemaps/ddb_topic_up_to_eight/sitemap-index.xml
Removed sitemap https://www.snapchat.com/sitemaps/extended_curated_topic/sitemap-index.xml

view diff → snapshot

Fox News foxnews.com modified +3 −1 2mo ago

Added Disallow: /api/article-search for *
Added Disallow: /xid for *
Removed sitemap https://video.foxnews.com/videositemap.xml

view diff → snapshot

First American Financial firstam.com modified +0 −1 2mo ago

Removed Allow: /ownership/what-is-escrow/ for *

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago

Added sitemap http://www.iqvia.com/image-sitemap.xml
Added sitemap http://www.iqvia.com/page-sitemap.xml
Removed sitemap http://127.0.0.1/image-sitemap.xml
Removed sitemap http://127.0.0.1/page-sitemap.xml

view diff → snapshot

DSV dsv.com modified +21 −21 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Nubank nubank.com.br modified +54 −54 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

WordPress wordpress.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

Qwant qwant.com modified +6 −2 2mo ago

Added Disallow: /?*bottomMenu=* for *
Added Disallow: /?*drawer=* for *
Added Disallow: /?*home=* for *
Removed Disallow: /account for *
🤖 AI Added Disallow: /?*bottomMenu=* for AI2Bot
🤖 AI Added Disallow: /?*drawer=* for AI2Bot
… 45 more change(s)

view diff → snapshot

Figma Community figma.com modified +7 −0 2mo ago

🤖 AI Added Allow: /de-de/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /es-es/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /es-la/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /fr-fr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /ja-jp/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
🤖 AI Added Allow: /ko-kr/blog/measuring-time-savings-from-figma-make/$ for CHATGPT-User
… 29 more change(s)

view diff → snapshot

Barracuda barracuda.com removed +0 −225 2mo ago

robots.txt is no longer served (HTTP 404)

Slate slate.com modified +5 −0 2mo ago

Added rules for user-agent ‘captify-crawler’
Added rules for user-agent ‘GnowitNewsbot’
Added rules for user-agent ‘peer39_crawler’
Blocked user-agent ‘Reflectionbot’ entirely
Added rules for user-agent ‘SirdataBot’

view diff → snapshot

Scale AI scale.com modified +256 −0 2mo ago

Added rules for user-agent ‘Discordbot’
Added rules for user-agent ‘facebookexternalhit’
Added rules for user-agent ‘LinkedInBot’
Added rules for user-agent ‘Slackbot’
Added rules for user-agent ‘Slackbot-LinkExpanding’
Added rules for user-agent ‘TelegramBot’
… 2 more change(s)

view diff → snapshot

Reuters reuters.com modified +1 −0 2mo ago

Added sitemap https://www.reuters.com/service/archive-sitemap/sitemap.xml

view diff → snapshot

Adidas adidas.com modified +1 −1 2mo ago

Formatting/comment-only change (no semantic difference)

view diff → snapshot

S&P Global spglobal.com modified +8 −0 2mo ago

Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/articles-sitemap-index.xml
Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/entities-sitemap-index.xml
Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/news-sitemap.xml
Added sitemap https://colombia.ratings.spglobal.com/ratings/es/sitemaps/sitemap.xml
Added sitemap https://panama.ratings.spglobal.com/ratings/es/sitemaps/articles-sitemap-index.xml
Added sitemap https://panama.ratings.spglobal.com/ratings/es/sitemaps/entities-sitemap-index.xml
… 2 more change(s)

view diff → snapshot

IQVIA Holdings iqvia.com modified +2 −2 2mo ago