Common Crawl / robots.txt
← back to commoncrawl.org · change detected 2mo ago (2026-08-17T11:01:05Z) · modified +7 −3
What changed
- Blocked user-agent ‘IndeedBot’ entirely
- Blocked user-agent ‘Indeedshopbot’ entirely
- Removed Allow: / for *
Diff old (2026-06-24T16:12:44Z) → new (2026-08-17T11:01:05Z)
| @@ -1,4 +1,4 @@ | ||
| 1 | - # All robots are explicitly allowed! | |
| 1 | + # We like well-behaved bots! | |
| 2 | 2 | |
| 3 | 3 | # |
| 4 | 4 | # ^~. |
| @@ -48,9 +48,13 @@ | ||
| 48 | 48 | #!GGBBB##########&######&###&&&&##&&&&&#&&&&&&&&&&&&&&&&&&&&&#&&&&&&&&&&&&&&&&&&&&&&&&&&&&& |
| 49 | 49 | #7GBBB##########&&&&&&#&&###&&&&&&&&&&##&&&&&&&&&&&&&&&&&&&&&##&&&&&&&&&&&&&&&&&&&&&&&&&&&& |
| 50 | 50 | |
| 51 | + User-agent: IndeedBot | |
| 52 | + Disallow: / | |
| 53 | + | |
| 54 | + User-agent: Indeedshopbot | |
| 55 | + Disallow: / | |
| 56 | + | |
| 51 | 57 | User-agent: * |
| 52 | - Allow: / | |
| 53 | 58 | Disallow: /search?* |
| 54 | 59 | |
| 55 | - | |
| 56 | 60 | Sitemap: https://commoncrawl.org/sitemap.xml |