Costa Rica (Presidencia) / robots.txt
← back to presidencia.go.cr · change detected 3d ago (2026-09-16T02:43:17Z) · modified +70 −59
What changed
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
- 🤖 AIUnblocked Applebot-Extended (Apple) - rules removed
- 🤖 AIUnblocked Bytespider (ByteDance) - rules removed
- 🤖 AIUnblocked CCBot (Common Crawl) - rules removed
- 🤖 AIUnblocked ClaudeBot (Anthropic) - rules removed
- Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
- 🤖 AIUnblocked Google-Extended (Google) - rules removed
- 🤖 AIUnblocked GPTBot (OpenAI) - rules removed
- 🤖 AIUnblocked meta-externalagent (Meta) - rules removed
- Added Allow: /core/*.css$ for *
- Added Allow: /core/*.css? for *
- Added Allow: /core/*.gif for *
- Added Allow: /core/*.jpeg for *
- Added Allow: /core/*.jpg for *
- Added Allow: /core/*.js$ for *
- Added Allow: /core/*.js? for *
- Added Allow: /core/*.png for *
- Added Allow: /core/*.svg for *
- Added Allow: /profiles/*.css$ for *
- Added Allow: /profiles/*.css? for *
- Added Allow: /profiles/*.gif for *
- Added Allow: /profiles/*.jpeg for *
- Added Allow: /profiles/*.jpg for *
- Added Allow: /profiles/*.js$ for *
- Added Allow: /profiles/*.js? for *
- Added Allow: /profiles/*.png for *
- Added Allow: /profiles/*.svg for *
- Added Disallow: /*/media/oembed for *
- Added Disallow: /README.md for *
- Added Disallow: /admin/ for *
- Added Disallow: /comment/reply/ for *
- Added Disallow: /composer/Metapackage/README.txt for *
- Added Disallow: /composer/Plugin/ProjectMessage/README.md for *
- Added Disallow: /composer/Plugin/Scaffold/README.md for *
- Added Disallow: /composer/Plugin/VendorHardening/README.txt for *
- Added Disallow: /composer/Template/README.txt for *
- Added Disallow: /core/ for *
- Added Disallow: /filter/tips for *
- Added Disallow: /index.php/*/media/oembed for *
- Added Disallow: /index.php/admin/ for *
- Added Disallow: /index.php/comment/reply/ for *
- Added Disallow: /index.php/filter/tips for *
- Added Disallow: /index.php/media/oembed for *
- Added Disallow: /index.php/node/add/ for *
- Added Disallow: /index.php/search/ for *
- Added Disallow: /index.php/user/login for *
- Added Disallow: /index.php/user/logout for *
- Added Disallow: /index.php/user/password for *
- Added Disallow: /index.php/user/register for *
- Added Disallow: /media/oembed for *
- ... and 11 more change(s)
Diff old (2026-09-11T19:15:58Z) → new (2026-09-16T02:43:17Z)
| @@ -1,61 +1,72 @@ | ||
| 1 | - # As a condition of accessing this website, you agree to abide by the following | |
| 2 | - # content signals: | |
| 3 | - | |
| 4 | - # (a) If a Content-Signal = yes, you may collect content for the corresponding | |
| 5 | - # use. | |
| 6 | - # (b) If a Content-Signal = no, you may not collect content for the | |
| 7 | - # corresponding use. | |
| 8 | - # (c) If the website operator does not include a Content-Signal for a | |
| 9 | - # corresponding use, the website operator neither grants nor restricts | |
| 10 | - # permission via Content-Signal with respect to the corresponding use. | |
| 11 | - | |
| 12 | - # The content signals and their meanings are: | |
| 13 | - | |
| 14 | - # search: building a search index and providing search results (e.g., returning | |
| 15 | - # hyperlinks and short excerpts from your website's contents). Search does not | |
| 16 | - # include providing AI-generated search summaries. | |
| 17 | - # ai-input: inputting content into one or more AI models (e.g., retrieval | |
| 18 | - # augmented generation, grounding, or other real-time taking of content for | |
| 19 | - # generative AI search answers). | |
| 20 | - # ai-train: training or fine-tuning AI models. | |
| 21 | - # use: how AI systems may consume the content (immediate, reference, or full). | |
| 22 | - | |
| 23 | - # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF | |
| 24 | - # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT | |
| 25 | - # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. | |
| 26 | - | |
| 27 | - # BEGIN Cloudflare Managed content | |
| 1 | + # | |
| 2 | + # robots.txt | |
| 3 | + # | |
| 4 | + # This file is to prevent the crawling and indexing of certain parts | |
| 5 | + # of your site by web crawlers and spiders run by sites like Yahoo! | |
| 6 | + # and Google. By telling these "robots" where not to go on your site, | |
| 7 | + # you save bandwidth and server resources. | |
| 8 | + # | |
| 9 | + # This file will be ignored unless it is at the root of your host: | |
| 10 | + # Used: http://example.com/robots.txt | |
| 11 | + # Ignored: http://example.com/site/robots.txt | |
| 12 | + # | |
| 13 | + # For more information about the robots.txt standard, see: | |
| 14 | + # http://www.robotstxt.org/robotstxt.html | |
| 28 | 15 | |
| 29 | 16 | User-agent: * |
| 30 | - Content-Signal: search=yes,ai-train=no,use=reference | |
| 31 | - Allow: / | |
| 32 | - | |
| 33 | - User-agent: Amazonbot | |
| 34 | - Disallow: / | |
| 35 | - | |
| 36 | - User-agent: Applebot-Extended | |
| 37 | - Disallow: / | |
| 38 | - | |
| 39 | - User-agent: Bytespider | |
| 40 | - Disallow: / | |
| 41 | - | |
| 42 | - User-agent: CCBot | |
| 43 | - Disallow: / | |
| 44 | - | |
| 45 | - User-agent: ClaudeBot | |
| 46 | - Disallow: / | |
| 47 | - | |
| 48 | - User-agent: CloudflareBrowserRenderingCrawler | |
| 49 | - Disallow: / | |
| 50 | - | |
| 51 | - User-agent: Google-Extended | |
| 52 | - Disallow: / | |
| 53 | - | |
| 54 | - User-agent: GPTBot | |
| 55 | - Disallow: / | |
| 56 | - | |
| 57 | - User-agent: meta-externalagent | |
| 58 | - Disallow: / | |
| 59 | - | |
| 60 | - # END Cloudflare Managed Content | |
| 61 | - | |
| 17 | + # CSS, JS, Images | |
| 18 | + Allow: /core/*.css$ | |
| 19 | + Allow: /core/*.css? | |
| 20 | + Allow: /core/*.js$ | |
| 21 | + Allow: /core/*.js? | |
| 22 | + Allow: /core/*.gif | |
| 23 | + Allow: /core/*.jpg | |
| 24 | + Allow: /core/*.jpeg | |
| 25 | + Allow: /core/*.png | |
| 26 | + Allow: /core/*.svg | |
| 27 | + Allow: /profiles/*.css$ | |
| 28 | + Allow: /profiles/*.css? | |
| 29 | + Allow: /profiles/*.js$ | |
| 30 | + Allow: /profiles/*.js? | |
| 31 | + Allow: /profiles/*.gif | |
| 32 | + Allow: /profiles/*.jpg | |
| 33 | + Allow: /profiles/*.jpeg | |
| 34 | + Allow: /profiles/*.png | |
| 35 | + Allow: /profiles/*.svg | |
| 36 | + # Directories | |
| 37 | + Disallow: /core/ | |
| 38 | + Disallow: /profiles/ | |
| 39 | + # Files | |
| 40 | + Disallow: /README.md | |
| 41 | + Disallow: /composer/Metapackage/README.txt | |
| 42 | + Disallow: /composer/Plugin/ProjectMessage/README.md | |
| 43 | + Disallow: /composer/Plugin/Scaffold/README.md | |
| 44 | + Disallow: /composer/Plugin/VendorHardening/README.txt | |
| 45 | + Disallow: /composer/Template/README.txt | |
| 46 | + Disallow: /modules/README.txt | |
| 47 | + Disallow: /sites/README.txt | |
| 48 | + Disallow: /themes/README.txt | |
| 49 | + # Paths (clean URLs) | |
| 50 | + Disallow: /admin/ | |
| 51 | + Disallow: /comment/reply/ | |
| 52 | + Disallow: /filter/tips | |
| 53 | + Disallow: /node/add/ | |
| 54 | + Disallow: /search/ | |
| 55 | + Disallow: /user/register | |
| 56 | + Disallow: /user/password | |
| 57 | + Disallow: /user/login | |
| 58 | + Disallow: /user/logout | |
| 59 | + Disallow: /media/oembed | |
| 60 | + Disallow: /*/media/oembed | |
| 61 | + # Paths (no clean URLs) | |
| 62 | + Disallow: /index.php/admin/ | |
| 63 | + Disallow: /index.php/comment/reply/ | |
| 64 | + Disallow: /index.php/filter/tips | |
| 65 | + Disallow: /index.php/node/add/ | |
| 66 | + Disallow: /index.php/search/ | |
| 67 | + Disallow: /index.php/user/password | |
| 68 | + Disallow: /index.php/user/register | |
| 69 | + Disallow: /index.php/user/login | |
| 70 | + Disallow: /index.php/user/logout | |
| 71 | + Disallow: /index.php/media/oembed | |
| 72 | + Disallow: /index.php/*/media/oembed |