NeuralCrawl

Vistra / robots.txt snapshot

← back to vistra.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 3065 bytes · sha256 4a9ef8fe16768292 · raw

final URL: https://www.vistra.com/robots.txt

1# https://www.vistra.com/robots.txt
2#
3# This file is to prevent the crawling and indexing of certain parts
4# of your site by web crawlers and spiders run by sites like Yahoo!
5# and Google. By telling these "robots" where not to go on your site,
6# you save bandwidth and server resources.
7#
8# This file will be ignored unless it is at the root of your host:
9# Used: https://www.vistra.com/robots.txt
10# Ignored: https://www.vistra.com/site/robots.txt
11#
12# For more information about the robots.txt standard, see:
13# http://www.robotstxt.org/robotstxt.html
14
15
16User-agent: Googlebot-News
17Disallow: /people/
18Disallow: /*/people/
19# Directories
20Disallow: /core/
21Disallow: /profiles/
22Disallow: /*/profiles/
23# Files
24Disallow: /README.txt
25Disallow: /web.config
26# Paths (clean URLs)
27Disallow: /admin/
28Disallow: /*/admin/
29Disallow: /comment/reply/
30Disallow: /*/comment/reply/
31Disallow: /filter/tips
32Disallow: /*/filter/tips
33Disallow: /node/ #PP changed from /node/add/
34Disallow: /*/node/
35Disallow: /search/? #PP added "?"
36Disallow: /*/search/?
37Disallow: /user/ #PP added
38Disallow: /*/user/
39Disallow: /user/register/
40Disallow: /*/user/register/
41Disallow: /user/password/
42Disallow: /*/user/password/
43Disallow: /user/login/
44Disallow: /*/user/login/
45Disallow: /user/logout/
46Disallow: /*/user/logout/
47# Paths (no clean URLs)
48Disallow: /term-listing/
49Disallow: /*/term-listing/
50Disallow: /print/
51Disallow: /*/print/
52#Disallow: /de/taxonomy/term/ #canonicalised so not deindexed
53#Disallow: /zh-hans/taxonomy/term/ #canonicalised so not deindexed
54# CSS, JS, Images
55Allow: /*.js
56Allow: /*.css
57Allow: /*.gif
58Allow: /*.jpeg
59Allow: /*.jpg
60Allow: /*.png
61Allow: /*.svg
62Allow: /*.woff2
63Allow: /*utm_campaign=google_search_box
64
65User-agent: *
66# Directories
67Disallow: /core/
68Disallow: /profiles/
69Disallow: /*/profiles/
70# Files
71Disallow: /README.txt
72Disallow: /web.config
73# Paths (clean URLs)
74Disallow: /admin/
75Disallow: /*/admin/
76Disallow: /comment/reply/
77Disallow: /*/comment/reply/
78Disallow: /filter/tips
79Disallow: /*/filter/tips
80Disallow: /node/ #PP changed from /node/add/
81Disallow: /*/node/
82Disallow: /search/? #PP added "?"
83Disallow: /*/search/?
84Disallow: /user/ #PP added
85Disallow: /*/user/
86Disallow: /user/register/
87Disallow: /*/user/register/
88Disallow: /user/password/
89Disallow: /*/user/password/
90Disallow: /user/login/
91Disallow: /*/user/login/
92Disallow: /user/logout/
93Disallow: /*/user/logout/
94# Paths (no clean URLs)
95Disallow: /term-listing/
96Disallow: /*/term-listing/
97Disallow: /print/
98Disallow: /*/print/
99#Disallow: /de/taxonomy/term/ #canonicalised so not deindexed
100#Disallow: /zh-hans/taxonomy/term/ #canonicalised so not deindexed
101# CSS, JS, Images
102Allow: /*.js
103Allow: /*.css
104Allow: /*.gif
105Allow: /*.jpeg
106Allow: /*.jpg
107Allow: /*.png
108Allow: /*.svg
109Allow: /*.woff2
110Allow: /*utm_campaign=google_search_box
111
112# BB Additional rules
113User-agent: meanpathbot
114User-agent: MegaIndex.ru/2.0
115User-agent: MegaIndex.ru/
116User-agent: AhrefsBot
117User-Agent: Sogou web spider
118User-Agent: Qwantify
119User-Agent: Barkrowler
120User-Agent: PetalBot
121Disallow: /
122
123
124Sitemap: https://www.vistra.com/sitemap.xml