NeuralCrawl

Vistra / robots.txt snapshot

← back to vistra.com · fetched 2026-08-27T20:19:04Z (22d ago) · HTTP 200 · 3233 bytes · sha256 cffa4c9a7bacc036 · raw

final URL: https://www.vistra.com/robots.txt

1# https://www.vistra.com/robots.txt
2#
3# This file is to prevent the crawling and indexing of certain parts
4# of your site by web crawlers and spiders run by sites like Yahoo!
5# and Google. By telling these "robots" where not to go on your site,
6# you save bandwidth and server resources.
7#
8# This file will be ignored unless it is at the root of your host:
9# Used: https://www.vistra.com/robots.txt
10# Ignored: https://www.vistra.com/site/robots.txt
11#
12# For more information about the robots.txt standard, see:
13# http://www.robotstxt.org/robotstxt.html
14
15
16User-agent: Googlebot-News
17Disallow: /people/
18Disallow: /*/people/
19# Directories
20Disallow: /core/
21Disallow: /profiles/
22Disallow: /*/profiles/
23# Files
24Disallow: /README.txt
25Disallow: /web.config
26# Paths (clean URLs)
27Disallow: /admin/
28Disallow: /*/admin/
29Disallow: /comment/reply/
30Disallow: /*/comment/reply/
31Disallow: /filter/tips
32Disallow: /*/filter/tips
33Disallow: /media/oembed #media embed endpoints, not pages
34Disallow: /*/media/oembed
35Disallow: /node/ #PP changed from /node/add/
36Disallow: /*/node/
37Disallow: /search/? #PP added "?"
38Disallow: /*/search/?
39Disallow: /user/ #PP added
40Disallow: /*/user/
41Disallow: /user/register/
42Disallow: /*/user/register/
43Disallow: /user/password/
44Disallow: /*/user/password/
45Disallow: /user/login/
46Disallow: /*/user/login/
47Disallow: /user/logout/
48Disallow: /*/user/logout/
49# Paths (no clean URLs)
50Disallow: /term-listing/
51Disallow: /*/term-listing/
52Disallow: /print/
53Disallow: /*/print/
54#Disallow: /de/taxonomy/term/ #canonicalised so not deindexed
55#Disallow: /zh-hans/taxonomy/term/ #canonicalised so not deindexed
56# CSS, JS, Images
57Allow: /*.js
58Allow: /*.css
59Allow: /*.gif
60Allow: /*.jpeg
61Allow: /*.jpg
62Allow: /*.png
63Allow: /*.svg
64Allow: /*.woff2
65Allow: /*utm_campaign=google_search_box
66
67User-agent: *
68# Directories
69Disallow: /core/
70Disallow: /profiles/
71Disallow: /*/profiles/
72# Files
73Disallow: /README.txt
74Disallow: /web.config
75# Paths (clean URLs)
76Disallow: /admin/
77Disallow: /*/admin/
78Disallow: /comment/reply/
79Disallow: /*/comment/reply/
80Disallow: /filter/tips
81Disallow: /*/filter/tips
82Disallow: /media/oembed #media embed endpoints, not pages
83Disallow: /*/media/oembed
84Disallow: /node/ #PP changed from /node/add/
85Disallow: /*/node/
86Disallow: /search/? #PP added "?"
87Disallow: /*/search/?
88Disallow: /user/ #PP added
89Disallow: /*/user/
90Disallow: /user/register/
91Disallow: /*/user/register/
92Disallow: /user/password/
93Disallow: /*/user/password/
94Disallow: /user/login/
95Disallow: /*/user/login/
96Disallow: /user/logout/
97Disallow: /*/user/logout/
98# Paths (no clean URLs)
99Disallow: /term-listing/
100Disallow: /*/term-listing/
101Disallow: /print/
102Disallow: /*/print/
103#Disallow: /de/taxonomy/term/ #canonicalised so not deindexed
104#Disallow: /zh-hans/taxonomy/term/ #canonicalised so not deindexed
105# CSS, JS, Images
106Allow: /*.js
107Allow: /*.css
108Allow: /*.gif
109Allow: /*.jpeg
110Allow: /*.jpg
111Allow: /*.png
112Allow: /*.svg
113Allow: /*.woff2
114Allow: /*utm_campaign=google_search_box
115
116# BB Additional rules
117User-agent: meanpathbot
118User-agent: MegaIndex.ru/2.0
119User-agent: MegaIndex.ru/
120User-agent: AhrefsBot
121User-Agent: Sogou web spider
122User-Agent: Qwantify
123User-Agent: Barkrowler
124User-Agent: PetalBot
125Disallow: /
126
127
128Sitemap: https://www.vistra.com/sitemap.xml