NeuralCrawl

Baxter International / robots.txt

← back to baxter.com · change detected 1mo ago (2026-09-21T10:05:12Z) · modified +2 −133

What changed

Diff old (2026-07-12T16:30:56Z) → new (2026-09-21T10:05:12Z)

@@ -1,134 +1,3 @@
1- # As a condition of accessing this website, you agree to abide by the following
2- # content signals:
3-
4- # (a) If a Content-Signal = yes, you may collect content for the corresponding
5- # use.
6- # (b) If a Content-Signal = no, you may not collect content for the
7- # corresponding use.
8- # (c) If the website operator does not include a Content-Signal for a
9- # corresponding use, the website operator neither grants nor restricts
10- # permission via Content-Signal with respect to the corresponding use.
11-
12- # The content signals and their meanings are:
13-
14- # search: building a search index and providing search results (e.g., returning
15- # hyperlinks and short excerpts from your website's contents). Search does not
16- # include providing AI-generated search summaries.
17- # ai-input: inputting content into one or more AI models (e.g., retrieval
18- # augmented generation, grounding, or other real-time taking of content for
19- # generative AI search answers).
20- # ai-train: training or fine-tuning AI models.
21- # use: how AI systems may consume the content (immediate, reference, or full).
22-
23- # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
24- # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
25- # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
26-
27- # BEGIN Cloudflare Managed content
28-
291  User-agent: *
30- Content-Signal: search=yes,ai-train=no,use=reference
31- Allow: /
32-
33- User-agent: Amazonbot
34- Disallow: /
35-
36- User-agent: Applebot-Extended
37- Disallow: /
38-
39- User-agent: Bytespider
40- Disallow: /
41-
42- User-agent: CCBot
43- Disallow: /
44-
45- User-agent: ClaudeBot
46- Disallow: /
47-
48- User-agent: CloudflareBrowserRenderingCrawler
49- Disallow: /
50-
51- User-agent: Google-Extended
52- Disallow: /
53-
54- User-agent: GPTBot
55- Disallow: /
56-
57- User-agent: meta-externalagent
58- Disallow: /
59-
60- # END Cloudflare Managed Content
61-
62- #
63- # robots.txt
64- #
65- # This file is to prevent the crawling and indexing of certain parts
66- # of your site by web crawlers and spiders run by sites like Yahoo!
67- # and Google. By telling these "robots" where not to go on your site,
68- # you save bandwidth and server resources.
69- #
70- # This file will be ignored unless it is at the root of your host:
71- # Used: http://example.com/robots.txt
72- # Ignored: http://example.com/site/robots.txt
73- #
74- # For more information about the robots.txt standard, see:
75- # http://www.robotstxt.org/robotstxt.html
76-
77- User-agent: *
78- # CSS, JS, Images
79- Allow: /core/*.css$
80- Allow: /core/*.css?
81- Allow: /core/*.js$
82- Allow: /core/*.js?
83- Allow: /core/*.gif
84- Allow: /core/*.jpg
85- Allow: /core/*.jpeg
86- Allow: /core/*.png
87- Allow: /core/*.svg
88- Allow: /profiles/*.css$
89- Allow: /profiles/*.css?
90- Allow: /profiles/*.js$
91- Allow: /profiles/*.js?
92- Allow: /profiles/*.gif
93- Allow: /profiles/*.jpg
94- Allow: /profiles/*.jpeg
95- Allow: /profiles/*.png
96- Allow: /profiles/*.svg
97- # Directories
98- Disallow: /core/
99- Disallow: /profiles/
100- # Files
101- Disallow: /README.md
102- Disallow: /composer/Metapackage/README.txt
103- Disallow: /composer/Plugin/ProjectMessage/README.md
104- Disallow: /composer/Plugin/Scaffold/README.md
105- Disallow: /composer/Plugin/VendorHardening/README.txt
106- Disallow: /composer/Template/README.txt
107- Disallow: /modules/README.txt
108- Disallow: /sites/README.txt
109- Disallow: /themes/README.txt
110- Disallow: /web.config
111- # Paths (clean URLs)
112- Disallow: /admin/
113- Disallow: /comment/reply/
114- Disallow: /filter/tips
115- Disallow: /node/add/
116- Disallow: /search/
117- Disallow: /user/register
118- Disallow: /user/password
119- Disallow: /user/login
120- Disallow: /user/logout
121- Disallow: /media/oembed
122- Disallow: /*/media/oembed
123- # Paths (no clean URLs)
124- Disallow: /index.php/admin/
125- Disallow: /index.php/comment/reply/
126- Disallow: /index.php/filter/tips
127- Disallow: /index.php/node/add/
128- Disallow: /index.php/search/
129- Disallow: /index.php/user/password
130- Disallow: /index.php/user/register
131- Disallow: /index.php/user/login
132- Disallow: /index.php/user/logout
133- Disallow: /index.php/media/oembed
134- Disallow: /index.php/*/media/oembed
2+ Disallow: /cdn-cgi/
3+ Sitemap: https://www.baxter.com/sitemap.xml