NeuralCrawl

Universities

The archive — every monitored site, its robots.txt history and AI-crawler stance.

All sites Top 1000 websites US 500 Nasdaq 100 AI models Cybersecurity SEO & AI search Creative rights-holders Knowledge & UGC Academic & open research Universities National indices Banks & fintech E-commerce European companies Social Networks Governments Publishers
# Company Domain Sector robots.txt AI bots blocked Snapshots Changes Last change
1 🇺🇸 Massachusetts Institute of Technology mit.edu University present 0 1 0 —
2 🇬🇧 University of Cambridge cam.ac.uk University present 0 1 0 —
3 🇬🇧 University of Oxford ox.ac.uk University blocked 0 0 0 —
4 🇺🇸 Harvard University harvard.edu University present 0 1 0 —
5 🇺🇸 Stanford University stanford.edu University present 0 1 0 —
6 🇺🇸 University of California Berkeley berkeley.edu University present 0 1 0 —
7 🇺🇸 California Institute of Technology caltech.edu University present 0 1 0 —
8 🇺🇸 Princeton University princeton.edu University present 0 1 0 —
9 🇺🇸 Yale University yale.edu University present 0 1 0 —
10 🇺🇸 Columbia University columbia.edu University absent 0 0 0 —
11 🇺🇸 University of Chicago uchicago.edu University present 0 1 0 —
12 🇺🇸 University of Pennsylvania upenn.edu University present 0 1 0 —
13 🇺🇸 Cornell University cornell.edu University present 0 1 0 —
14 🇺🇸 University of California Los Angeles ucla.edu University present 0 1 0 —
15 🇺🇸 University of Michigan umich.edu University present 0 1 0 —
16 🇬🇧 Imperial College London imperial.ac.uk University present 0 1 0 —
17 🇬🇧 University College London ucl.ac.uk University present 0 1 0 —
18 🇬🇧 University of Edinburgh ed.ac.uk University present 0 1 0 —
19 🇨🇭 ETH Zurich ethz.ch University absent 0 0 0 —
20 🇨🇭 EPFL epfl.ch University present 🤖 8 1 0 —
21 🇩🇪 Technical University of Munich tum.de University present 0 1 0 —
22 🇩🇪 Ludwig Maximilian University of Munich lmu.de University present 0 1 0 —
23 🇫🇷 Sorbonne University sorbonne-universite.fr University present 0 1 0 —
24 🇳🇱 University of Amsterdam uva.nl University present 0 1 0 —
25 🇸🇪 KTH Royal Institute of Technology kth.se University present 0 1 0 —
26 🇯🇵 University of Tokyo u-tokyo.ac.jp University present 0 1 0 —
27 🇯🇵 Kyoto University kyoto-u.ac.jp University present 0 1 0 —
28 🇸🇬 National University of Singapore nus.edu.sg University Not robots.txt 0 0 0 —
29 🇸🇬 Nanyang Technological University ntu.edu.sg University present 0 1 0 —
30 🇨🇳 Tsinghua University tsinghua.edu.cn University absent 0 0 0 —
31 🇨🇳 Peking University pku.edu.cn University absent 0 0 0 —
32 🇨🇳 Fudan University fudan.edu.cn University absent 0 0 0 —
33 🇦🇺 University of Melbourne unimelb.edu.au University blocked 0 0 0 —
34 🇦🇺 University of Sydney sydney.edu.au University present 0 1 0 —
35 🇮🇳 Indian Institute of Technology Bombay iitb.ac.in University present 0 1 0 —
36 🇮🇳 Indian Institute of Science iisc.ac.in University present 0 1 0 —
37 🇨🇦 University of Toronto utoronto.ca University present 0 1 0 —
38 🇨🇦 University of British Columbia ubc.ca University present 0 1 0 —
39 🇨🇦 McGill University mcgill.ca University unreachable 0 0 0 —
40 🇧🇷 University of São Paulo usp.br University redirect 0 0 0 —
41 🇲🇽 National Autonomous University of Mexico unam.mx University blocked 0 0 0 —
42 🇸🇦 King Abdullah University of Science and Technology kaust.edu.sa University redirect 0 0 0 —
43 🇮🇱 Tel Aviv University tau.ac.il University present 0 1 0 —
44 🇿🇦 University of Cape Town uct.ac.za University present 🤖 3 1 0 —