🇬🇧 Imperial College London

AI crawler access (latest snapshot, 3h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

Current robots.txt 6439 bytes · sha256 42325c7ea865 · raw

User-agent: slurp
Crawl-delay: 0.5

User-agent: *
Disallow: /terminalfour/SiteManager/
Disallow: /business-school/icbs-apis/
# CSS, JS, Images
Allow: /business-school/core/*.css$
Allow: /business-school/core/*.css?
Allow: /business-school/core/*.js$
Allow: /business-school/core/*.js?
Allow: /business-school/core/*.gif
Allow: /business-school/core/*.jpg
Allow: /business-school/core/*.jpeg
Allow: /business-school/core/*.png
Allow: /business-school/core/*.svg
Allow: /business-school/profiles/*.css$
Allow: /business-school/profiles/*.css?
Allow: /business-school/profiles/*.js$
Allow: /business-school/profiles/*.js?
Allow: /business-school/profiles/*.gif
Allow: /business-school/profiles/*.jpg
Allow: /business-school/profiles/*.jpeg
Allow: /business-school/profiles/*.png
Allow: /business-school/profiles/*.svg
# Directories
Disallow: /business-school/core/
Disallow: /business-school/profiles/
# Files
Disallow: /business-school/README.txt
Disallow: /business-school/web.config
# Paths (clean URLs)
Disallow: /business-school/admin/
Disallow: /business-school/comment/reply/
Disallow: /business-school/filter/tips
Disallow: /business-school/node/add/
Disallow: /business-school/search/
Disallow: /business-school/user/register/
Disallow: /business-school/user/password/
Disallow: /business-school/user/login/
Disallow: /business-school/user/logout/

# Paths (no clean URLs)
Disallow: /business-school/index.php/admin/
Disallow: /business-school/index.php/comment/reply/
Disallow: /business-school/index.php/filter/tips
Disallow: /business-school/index.php/node/add/
Disallow: /business-school/index.php/search/
Disallow: /business-school/index.php/user/password/
Disallow: /business-school/index.php/user/register/
Disallow: /business-school/index.php/user/login/
Disallow: /business-school/index.php/user/logout/

# business school thank you pages and others
Disallow: /business-school/phd/mailing-list/thank-you/
Disallow: /business-school/phd/brochure/thank-you/
Disallow: /business-school/undergraduate/summer-school/mailing-list/thank-you/ 
Disallow: /business-school/undergraduate/summer-school/brochure/thank-you/ 
Disallow: /business-school/undergraduate/mailing-list/thank-you 
Disallow: /business-school/undergraduate/brochure/thank-you 
Disallow: /business-school/masters/mailing-list/thank-you 
Disallow: /business-school/masters/brochure/thank-you 
Disallow: /business-school/mba/full-time-mba/class-profile/thank-you/ 
Disallow: /business-school/mba/full-time-mba/career-impact/thank-you/ 
Disallow: /business-school/mba/executive-mba/class-profile/thank-you/ 
Disallow: /business-school/mba/executive-mba/consultation/thank-you/ 
Disallow: /business-school/mba/weekend-mba/career-impact/thank-you/ 
Disallow: /business-school/mba/weekend-mba/class-profile/thank-you/ 
Disallow: /business-school/mba/weekend-mba/consultation/thank-you/ 
Disallow: /business-school/mba/global-online-mba/consultation/thank-you/ 
Disallow: /business-school/mba/global-online-mba/career-impact/thank-you/ 
Disallow: /business-school/mba/brochure/thank-you 
Disallow: /business-school/mba/mailing-list/thank-you 
Disallow: /business-school/programmes/gmat-waiver/thank-you
Disallow: /business-school/masters/finance-employment-report-thank-you
Disallow: /business-school/masters/international-management/career-impact/thank-you/ 
Disallow: /business-school/masters/international-management/consultation/thank-you/ 
Disallow: /business-school/programmes/thank-you-for-registering/ 
Disallow: /business-school/executive-education/virtual-programmes/request-call-back/thank-you/
Disallow: /business-school/executive-education/staff-training-and-development/thank-you/
Disallow: /business-school/executive-education/download-virtual-brochure/thank-you/ 
Disallow: /business-school/executive-education/download-brochure/thank-you/ 
Disallow: /business-school/executive-education/download-calendar/thank-you/ 
Disallow: /business-school/executive-education/alumni-discount/thank-you/ 
Disallow: /business-school/executive-education/application-form/group-bookings/thank-you/ 
Disallow: /business-school/executive-education/request-call-back/thank-you/ 
Disallow: /business-school/executive-education/application-form/virtual-programmes/thank-you/ 
Disallow: /business-school/executive-education/custom-programmes/closed-group-programme-enquiry/thank-you/ 
Disallow: /business-school/executive-education/events/webinar-thank-you/ 
Disallow: /business-school/executive-education/contact-us/customised-programmes-enquiries/thank-you/ 
Disallow: /business-school/executive-education/contact-us/short-courses-enquiry-form/thank-you/ 
Disallow: /business-school/executive-education/events/sign-up/thank-you/ 
Disallow: /business-school/executive-education/application-form/thank-you/ 
Disallow: /business-school/news-and-events/sign-news-and-events/thank-you/ 
Disallow: /business-school/events/webinar-thank-you-for-registering/ 
Disallow: /business-school/corporate-relations/recruit-new-talent/request-student-cvs-and-profile-books/thank-you/ 
Disallow: /business-school/sign-ib-knowledge/thank-you/
Disallow: /business-school/programmes/one-to-one-thank-you/

# search
Disallow: /search/

# old courses, update yearly
Disallow: /study/ug/courses/archived/
Disallow: /study/pg/courses/2018/
Disallow: /study/courses/undergraduate/2023/
Disallow: /study/courses/postgraduate-taught/2023/
Disallow: /study/courses/undergraduate/2024/
Disallow: /study/courses/postgraduate-taught/2024/
Disallow: /study/courses/undergraduate/2025/
Disallow: /study/courses/postgraduate-taught/2025/
Disallow: /study/courses/undergraduate/2026/

# old open days
Disallow: /study/visit/undergraduate/on-demand/

# chat to graduates
Disallow: /study/chat-to-our-graduates/

# Campaigns WIP
Disallow: /campaigns/

# development sites and redesign content branches
Disallow: /dev-sites/
Disallow: /redesign-content/
Disallow: /website-testing/
Disallow: /ib-redesign/

# reporter
Disallow: /staff/reporter/

#Academic Strategy
Disallow: /academic-strategy/

#Welcome week
Disallow: /students/new-students/welcome/

#News redesign and test
Disallow: /redesign-news/
Disallow: /news/training/

#College directory
Disallow: /collegedirectory/

#Business School redesign
Disallow: /ib-redesign/

# Sitemap files
Sitemap: https://www.imperial.ac.uk/sitemap.xml
Sitemap: https://www.imperial.ac.uk/business-school/sitemap.xml
Sitemap: https://www.imperial.ac.uk/people/sitemap.xml

Change history

1. initial 3h ago snapshot

   • First snapshot of robots.txt archived