NeuralCrawl

Costa Rica (Presidencia) / robots.txt

← back to presidencia.go.cr · change detected 15d ago (2026-10-22T21:05:52Z) · modified +128 −70

What changed

Diff old (2026-09-16T02:43:17Z) → new (2026-10-22T21:05:52Z)

@@ -1,72 +1,130 @@
1- #
2- # robots.txt
3- #
4- # This file is to prevent the crawling and indexing of certain parts
5- # of your site by web crawlers and spiders run by sites like Yahoo!
6- # and Google. By telling these "robots" where not to go on your site,
7- # you save bandwidth and server resources.
8- #
9- # This file will be ignored unless it is at the root of your host:
10- # Used: http://example.com/robots.txt
11- # Ignored: http://example.com/site/robots.txt
12- #
13- # For more information about the robots.txt standard, see:
14- # http://www.robotstxt.org/robotstxt.html
1+ # As a condition of accessing this website, you agree to abide by the following
2+ # content signals:
3+
4+ # (a) If a Content-Signal = yes, you may collect content for the corresponding
5+ # use.
6+ # (b) If a Content-Signal = no, you may not collect content for the
7+ # corresponding use.
8+ # (c) If the website operator does not include a Content-Signal for a
9+ # corresponding use, the website operator neither grants nor restricts
10+ # permission via Content-Signal with respect to the corresponding use.
11+
12+ # The content signals and their meanings are:
13+
14+ # search: building a search index and providing search results (e.g., returning
15+ # hyperlinks and short excerpts from your website's contents). Search does not
16+ # include providing AI-generated search summaries.
17+ # ai-input: inputting content into one or more AI models (e.g., retrieval
18+ # augmented generation, grounding, or other real-time taking of content for
19+ # generative AI search answers).
20+ # ai-train: training or fine-tuning AI models.
21+ # use: how AI systems may consume the content (immediate, reference, or full).
22+
23+ # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24+ # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25+ # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26+
27+ # BEGIN Cloudflare Managed content
28+
1529 
1630  User-agent: *
17- # CSS, JS, Images
18- Allow: /core/*.css$
19- Allow: /core/*.css?
20- Allow: /core/*.js$
21- Allow: /core/*.js?
22- Allow: /core/*.gif
23- Allow: /core/*.jpg
24- Allow: /core/*.jpeg
25- Allow: /core/*.png
26- Allow: /core/*.svg
27- Allow: /profiles/*.css$
28- Allow: /profiles/*.css?
29- Allow: /profiles/*.js$
30- Allow: /profiles/*.js?
31- Allow: /profiles/*.gif
32- Allow: /profiles/*.jpg
33- Allow: /profiles/*.jpeg
34- Allow: /profiles/*.png
35- Allow: /profiles/*.svg
36- # Directories
37- Disallow: /core/
38- Disallow: /profiles/
39- # Files
40- Disallow: /README.md
41- Disallow: /composer/Metapackage/README.txt
42- Disallow: /composer/Plugin/ProjectMessage/README.md
43- Disallow: /composer/Plugin/Scaffold/README.md
44- Disallow: /composer/Plugin/VendorHardening/README.txt
45- Disallow: /composer/Template/README.txt
46- Disallow: /modules/README.txt
47- Disallow: /sites/README.txt
48- Disallow: /themes/README.txt
49- # Paths (clean URLs)
50- Disallow: /admin/
51- Disallow: /comment/reply/
52- Disallow: /filter/tips
53- Disallow: /node/add/
54- Disallow: /search/
55- Disallow: /user/register
56- Disallow: /user/password
57- Disallow: /user/login
58- Disallow: /user/logout
59- Disallow: /media/oembed
60- Disallow: /*/media/oembed
61- # Paths (no clean URLs)
62- Disallow: /index.php/admin/
63- Disallow: /index.php/comment/reply/
64- Disallow: /index.php/filter/tips
65- Disallow: /index.php/node/add/
66- Disallow: /index.php/search/
67- Disallow: /index.php/user/password
68- Disallow: /index.php/user/register
69- Disallow: /index.php/user/login
70- Disallow: /index.php/user/logout
71- Disallow: /index.php/media/oembed
72- Disallow: /index.php/*/media/oembed
31+ Content-Signal: search=yes,ai-train=no,use=reference
32+ Allow: /
33+
34+ # Training crawlers
35+ User-agent: Amazonbot
36+ Disallow: /
37+
38+ User-agent: Applebot-Extended
39+ Disallow: /
40+
41+ User-agent: Bytespider
42+ Disallow: /
43+
44+ User-agent: CCBot
45+ Disallow: /
46+
47+ User-agent: ClaudeBot
48+ Disallow: /
49+
50+ User-agent: Diffbot
51+ Disallow: /
52+
53+ User-agent: Google-Extended
54+ Disallow: /
55+
56+ User-agent: GPTBot
57+ Disallow: /
58+
59+ User-agent: omgili
60+ Disallow: /
61+
62+ User-agent: anthropic-ai
63+ Disallow: /
64+
65+ User-agent: Claude-Web
66+ Disallow: /
67+
68+ User-agent: cohere-ai
69+ Disallow: /
70+
71+ User-agent: MistralAI-Training
72+ Disallow: /
73+
74+ User-agent: meta-externalagent
75+ Disallow: /
76+
77+ User-agent: GoogleOther
78+ Disallow: /
79+
80+ User-agent: Baiduspider
81+ Disallow: /
82+
83+ User-agent: PetalBot
84+ Disallow: /
85+
86+ User-agent: AwarioSmartBot
87+ User-agent: AwarioRssBot
88+ Disallow: /
89+
90+ User-agent: Google-CloudVertexBot
91+ Disallow: /
92+
93+ User-agent: QualifiedBot
94+ Disallow: /
95+
96+ User-agent: Cotoyogi
97+ Disallow: /
98+
99+ User-agent: ICC-Crawler
100+ Disallow: /
101+
102+ User-agent: atlassian-bot
103+ Disallow: /
104+
105+ User-agent: FishBot
106+ Disallow: /
107+
108+ User-agent: BorderxBot
109+ Disallow: /
110+
111+ User-agent: NavuBot
112+ Disallow: /
113+
114+ User-agent: SemrushBot-SWA
115+ Disallow: /
116+
117+ User-agent: WARDBot
118+ Disallow: /
119+
120+ User-agent: magpie-crawler
121+ Disallow: /
122+
123+ User-agent: KimiBot
124+ Disallow: /
125+
126+ User-agent: CitibotSiteCrawler
127+ Disallow: /
128+
129+ # END Cloudflare Managed Content
130+