NeuralCrawl

Universities

The archive — every monitored site, its robots.txt history and AI-crawler stance.

All sites Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers
# Company Domain Sector robots.txt AI bots blocked Snapshots Changes Last change
1 🇺🇸 Massachusetts Institute of Technology mit.edu University present 0 1 0 —
2 🇬🇧 University of Cambridge cam.ac.uk University present 0 1 0 —
3 🇬🇧 University of Oxford ox.ac.uk University present 0 1 0 —
4 🇺🇸 Harvard University harvard.edu University present 0 1 0 —
5 🇺🇸 Stanford University stanford.edu University present 0 1 0 —
6 🇺🇸 University of California Berkeley berkeley.edu University present 0 1 0 —
7 🇺🇸 California Institute of Technology caltech.edu University present 0 1 0 —
8 🇺🇸 Princeton University princeton.edu University present 0 1 0 —
9 🇺🇸 Yale University yale.edu University present 0 1 0 —
10 🇺🇸 Columbia University columbia.edu University absent 0 0 0 —
11 🇺🇸 University of Chicago uchicago.edu University present 0 1 0 —
12 🇺🇸 University of Pennsylvania upenn.edu University present 0 2 1 14d ago
13 🇺🇸 Cornell University cornell.edu University present 0 1 0 —
14 🇺🇸 University of California Los Angeles ucla.edu University present 0 1 0 —
15 🇺🇸 University of Michigan umich.edu University present 0 1 0 —
16 🇬🇧 Imperial College London imperial.ac.uk University present 0 1 0 —
17 🇬🇧 University College London ucl.ac.uk University present 0 2 1 19d ago
18 🇬🇧 University of Edinburgh ed.ac.uk University present 0 1 0 —
19 🇨🇭 ETH Zurich ethz.ch University absent 0 0 0 —
20 🇨🇭 EPFL epfl.ch University present 🤖 8 2 1 1mo ago
21 🇩🇪 Technical University of Munich tum.de University present 0 1 0 —
22 🇩🇪 Ludwig Maximilian University of Munich lmu.de University present 0 2 1 1mo ago
23 🇫🇷 Sorbonne University sorbonne-universite.fr University present 0 1 0 —
24 🇳🇱 University of Amsterdam uva.nl University present 0 1 0 —
25 🇸🇪 KTH Royal Institute of Technology kth.se University present 0 1 0 —
26 🇯🇵 University of Tokyo u-tokyo.ac.jp University present 0 1 0 —
27 🇯🇵 Kyoto University kyoto-u.ac.jp University present 0 1 0 —
28 🇸🇬 National University of Singapore nus.edu.sg University Not robots.txt 0 0 0 —
29 🇸🇬 Nanyang Technological University ntu.edu.sg University present 0 1 0 —
30 🇦🇺 University of Melbourne unimelb.edu.au University present 0 1 0 —
31 🇦🇺 University of Sydney sydney.edu.au University present 0 1 0 —
32 🇮🇳 Indian Institute of Technology Bombay iitb.ac.in University present 0 1 0 —
33 🇮🇳 Indian Institute of Science iisc.ac.in University present 0 1 0 —
34 🇨🇦 University of Toronto utoronto.ca University present 0 1 0 —
35 🇨🇦 University of British Columbia ubc.ca University present 0 1 0 —
36 🇨🇦 McGill University mcgill.ca University present 0 1 0 —
37 🇮🇱 Hebrew University of Jerusalem huji.ac.il University absent 0 0 0 —
38 🇮🇱 Technion int.technion.ac.il University present 0 1 0 —
39 🇮🇱 Tel Aviv University tau.ac.il University present 0 1 0 —
40 🇮🇱 Weizmann Institute of Science weizmann.ac.il University present 0 1 0 —
41 🇮🇱 Ben-Gurion University of the Negev bgu.ac.il University present 0 2 1 25d ago
42 🇮🇱 Bar-Ilan University biu.ac.il University present 0 1 0 —
43 🇮🇱 University of Haifa haifa.ac.il University present 0 1 0 —
44 🇮🇱 Reichman University runi.ac.il University present 0 1 0 —
45 🇿🇦 University of Cape Town uct.ac.za University present 🤖 2 3 2 1mo ago