Peru (Gob.pe) / robots.txt
← back to gob.pe · change detected 2mo ago (2026-08-15T04:24:46Z) · modified +14 −0
What changed
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- Added Disallow: /*&sheet= for *
- Added Disallow: /*?sheet= for *
Diff old (2026-06-20T01:10:30Z) → new (2026-08-15T04:24:46Z)
| @@ -3,8 +3,22 @@ | ||
| 3 | 3 | # To ban all spiders from the entire site uncomment the next two lines: |
| 4 | 4 | User-agent: * |
| 5 | 5 | Disallow: /admin/ |
| 6 | + Disallow: /*?sheet= | |
| 7 | + Disallow: /*&sheet= | |
| 6 | 8 | |
| 7 | 9 | User-agent: AhrefsBot |
| 8 | 10 | Disallow: / |
| 9 | 11 | |
| 12 | + User-agent: GPTBot | |
| 13 | + Crawl-delay: 5 | |
| 14 | + Disallow: /admin/ | |
| 15 | + Disallow: /*?sheet= | |
| 16 | + Disallow: /*&sheet= | |
| 17 | + | |
| 18 | + User-agent: OAI-SearchBot | |
| 19 | + Crawl-delay: 2 | |
| 20 | + Disallow: /admin/ | |
| 21 | + Disallow: /*?sheet= | |
| 22 | + Disallow: /*&sheet= | |
| 23 | + | |
| 10 | 24 | Sitemap: https://www.gob.pe/sitemaps/sitemap.xml.gz |