University of Cape Town / robots.txt
← back to uct.ac.za · change detected 1mo ago (2026-07-01T12:24:12Z) · modified +24 −2
What changed
- Added Disallow: /*/search for Bingbot
- Added Disallow: /*/search$ for Bingbot
- Added Disallow: /*/search/ for Bingbot
- Added Disallow: /*/search? for Bingbot
- Added Disallow: /*?*search= for Bingbot
- Added Disallow: /admin/ for Bingbot
- Added Disallow: /cgi-bin/ for Bingbot
- Added Disallow: /private/ for Bingbot
- Added Disallow: /search for Bingbot
- Added Disallow: /search/ for Bingbot
- Added Disallow: /secret.html for Bingbot
- Added Disallow: /tmp/ for Bingbot
- Removed Disallow: (empty) for Bingbot
- 🤖 AIAdded Disallow: /*/search for OAI-SearchBot
- 🤖 AIAdded Disallow: /*/search$ for OAI-SearchBot
- 🤖 AIAdded Disallow: /*/search/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /*/search? for OAI-SearchBot
- 🤖 AIAdded Disallow: /*?*search= for OAI-SearchBot
- 🤖 AIAdded Disallow: /admin/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /cgi-bin/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /private/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /search for OAI-SearchBot
- 🤖 AIAdded Disallow: /search/ for OAI-SearchBot
- 🤖 AIAdded Disallow: /secret.html for OAI-SearchBot
- 🤖 AIAdded Disallow: /tmp/ for OAI-SearchBot
- 🤖 AIRemoved Disallow: (empty) for OAI-SearchBot
Diff old (2026-06-29T11:44:49Z) → new (2026-07-01T12:24:12Z)
| @@ -72,7 +72,18 @@ | ||
| 72 | 72 | Disallow: /*?*search= |
| 73 | 73 | |
| 74 | 74 | User-agent: Bingbot |
| 75 | - Disallow: | |
| 75 | + Disallow: /private/ | |
| 76 | + Disallow: /secret.html | |
| 77 | + Disallow: /tmp/ | |
| 78 | + Disallow: /cgi-bin/ | |
| 79 | + Disallow: /admin/ | |
| 80 | + Disallow: /search | |
| 81 | + Disallow: /search/ | |
| 82 | + Disallow: /*/search | |
| 83 | + Disallow: /*/search/ | |
| 84 | + Disallow: /*/search$ | |
| 85 | + Disallow: /*/search? | |
| 86 | + Disallow: /*?*search= | |
| 76 | 87 | |
| 77 | 88 | User-agent: Slurp |
| 78 | 89 | Disallow: /private/ |
| @@ -194,7 +205,18 @@ | ||
| 194 | 205 | Disallow: / |
| 195 | 206 | |
| 196 | 207 | User-agent: OAI-SearchBot |
| 197 | - Disallow: | |
| 208 | + Disallow: /private/ | |
| 209 | + Disallow: /secret.html | |
| 210 | + Disallow: /tmp/ | |
| 211 | + Disallow: /cgi-bin/ | |
| 212 | + Disallow: /admin/ | |
| 213 | + Disallow: /search | |
| 214 | + Disallow: /search/ | |
| 215 | + Disallow: /*/search | |
| 216 | + Disallow: /*/search/ | |
| 217 | + Disallow: /*/search$ | |
| 218 | + Disallow: /*/search? | |
| 219 | + Disallow: /*?*search= | |
| 198 | 220 | |
| 199 | 221 | # sitemap.xml |
| 200 | 222 | Sitemap: https://uct.ac.za/sitemap.xml |