🇮🇱 The Times of Israel

timesofisrael.com · Publishers · rank #66 · News · live robots.txt ↗

AI crawler access (latest snapshot, 3h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅Meta-WebIndexer

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 1118 bytes · sha256 749720cb45cc · raw

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# digg mirror
User-agent: duggmirror
Disallow: /

# Twitter
User-agent: Twitterbot
Disallow:
Allow: /*

# Google News
User-agent: Googlebot-News
Disallow: /announcements/

# MSN
User-agent: msnbot
Crawl-delay: 3

User-agent: weborama-fetcher
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: 360Spider
Disallow: /

User-agent: YisouSpider
Disallow: /

User-agent: DotBot
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-includes/
Disallow: /wp-content/cache/
Disallow: /trackback/
Disallow: /rss-feed/
Disallow: /feed/
Disallow: /comments/
Disallow: /cities/
Disallow: /types/
Disallow: /anniversaries/
Disallow: /companies/
Disallow: /test-hp/
Disallow: *preview=true*
Disallow: /?p=*
Disallow: *3933714/TOI*
Disallow: /wp-content/themes/rgb/ajax/
Disallow: /wpcontent/themes/rbg/functions/

Sitemap: https://www.timesofisrael.com/sitemap_index.xml
Sitemap: https://www.timesofisrael.com/news-sitemap.xml

Change history

initial 3h ago snapshot
- First snapshot of robots.txt archived