🇮🇱 Weizmann Institute of Science

weizmann.ac.il · Universities · rank #40 · University · live robots.txt ↗

AI crawler access (latest snapshot, 2h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 2104 bytes · sha256 ff400b54028d · raw

# robots.txt for http://www.weizmann.ac.il

User-agent: *
Crawl-Delay: 10
Disallow: /DIS/
Disallow: /lsc/find-laboratories
Disallow: /HR/jobs/he/apply/
Disallow: /HR/jobs/en/apply/
Disallow: /old_pages/
Disallow: /molgen/Arama_old/
Disallow: /oldfeinberg/
Disallow: /conferences/Ilanit2011/
Disallow: /resdev/galleries/
Disallow: /conferences/Ilanit2014/
Disallow: /particle/milgrom/
Disallow: /Organic_Chemistry/milstein/
Disallow: /RGP/proposal/
Disallow: /RGP/proposals/
Disallow: /matlab/
Disallow: /cgi-bin/
Disallow: /perl/
Disallow: /CC/bulletin/
Disallow: /chemphys/tmg/
Disallow: /committees/donors/
Disallow: /committees/proposals/
Disallow: /committees/props/
Disallow: /conferences/Physics2Life/titles
Disallow: /conferences/Physics2Life/uploads/file/titles.pdf
Disallow: /davidson1/
Disallow: /davidson/admin/
Disallow: /davidson/barcode/
Disallow: /davidson/menu_js/
Disallow: /davidson/phpAdmin/
Disallow: /davidson/upload_music/
Disallow: /davidson/upload/
Disallow: /davidson/weather/
Disallow: /diff_angle/
Disallow: /feinberg2/
Disallow: /festival/
Disallow: /garden/
Disallow: /IU/create/
Disallow: /mediawiki/biology
Disallow: /neurobiology/labs/ahissar/group
Disallow: /photos
Disallow: /pipermail
Disallow: /usersfiles/
Disallow: /zemed/
User-agent: gsa-weizmann-crawler
Disallow: /cgi-bin/
Disallow: /perl/
Disallow: /chemphys/tmg/
Disallow: /committees/donors/
Disallow: /committees/proposals/
Disallow: /committees/props/
Disallow: /conferences/Physics2Life/titles
Disallow: /conferences/Physics2Life/uploads/file/titles.pdf
Disallow: /test
Disallow: /davidson/admin/
Disallow: /davidson/barcode/
Disallow: /davidson/menu_js/
Disallow: /davidson/phpAdmin/
Disallow: /davidson/upload_music/
Disallow: /davidson/upload/
Disallow: /davidson/weather/
Disallow: /diff_angle/
Disallow: /festival/
Disallow: /garden/
Disallow: /mediawiki/biology
Disallow: /neurobiology/labs/ahissar/group
Disallow: /photos
Disallow: /pipermail
Disallow: /usersfiles/
Disallow: /Web_Portfolio/
Disallow: /zemed/
Disallow: /midrasha/WomenInScience/
Disallow: /chemphys/idcards/KaterinaDubrovin4938.html

Change history

initial 2h ago snapshot
- First snapshot of robots.txt archived