NeuralCrawl

Mastercard / robots.txt snapshot

← back to mastercard.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 3554 bytes · sha256 381974e16128f09a · raw

final URL: https://www.mastercard.com/robots.txt

1User-agent: *
2Allow: /
3
4User-agent: *
5Disallow: /language-masters/
6
7User-agent: *
8Disallow: /admin/
9
10User-agent: *
11Disallow: /content/dam/mccom/protected/
12
13Sitemap: https://www.mastercard.com/sitemap-index.xml
14
15# Blocked Bot Requests
16
17User-agent: EmailCollector
18Disallow: /
19
20User-agent: EmailWolf
21Disallow: /
22
23User-agent: CopyRightCheck
24Disallow: /
25
26User-agent: CherryPicker
27Disallow: /
28
29User-agent: EmailSiphon
30Disallow: /
31
32User-agent: WebBandit
33Disallow: /
34
35User-agent: Crescent
36Disallow: /
37
38User-agent: NICErsPRO
39Disallow: /
40
41User-agent: SiteSnagger
42Disallow: /
43
44User-agent: ProWebWalker
45Disallow: /
46
47User-agent: CheeseBot
48Disallow: /
49
50User-agent: ia_archiver
51Disallow: /
52
53User-agent: ia_archiver/1.6
54Disallow: /
55
56User-agent: Teleport
57Disallow: /
58
59User-agent: TeleportPro
60Disallow: /
61
62User-agent: Wget
63Disallow: /
64
65User-agent: MIIxpc
66Disallow: /
67
68User-agent: Telesoft
69Disallow: /
70
71User-agent: Website Quester
72Disallow: /
73
74User-agent: WebZip
75Disallow: /
76
77User-agent: moget/2.1
78Disallow: /
79
80User-agent: WebZip/4.0
81Disallow: /
82
83User-agent: Mister PiX
84Disallow: /
85
86User-agent: WebStripper
87Disallow: /
88
89User-agent: WebSauger
90Disallow: /
91
92User-agent: WebCopier
93Disallow: /
94
95User-agent: NetAnts
96Disallow: /
97
98User-agent: WebAuto
99Disallow: /
100
101User-agent: TheNomad
102Disallow: /
103
104User-agent: WWW-Collector-E
105Disallow: /
106
107User-agent: RMA
108Disallow: /
109
110User-agent: libWeb/clsHTTP
111Disallow: /
112
113User-agent: asterias
114Disallow: /
115
116User-agent: httplib
117Disallow: /
118
119User-agent: turingos
120Disallow: /
121
122User-agent: spanner
123Disallow: /
124
125User-agent: InfoNaviRobot
126Disallow: /
127
128User-agent: Harvest/1.5
129Disallow: /
130
131User-agent: Bullseye/1.0
132Disallow: /
133
134User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)
135Disallow: /
136
137User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
138Disallow: /
139
140User-agent: CherryPickerSE/1.0
141Disallow: /
142
143User-agent: CherryPickerElite/1.0
144Disallow: /
145
146User-agent: WebBandit/3.50
147Disallow: /
148
149User-agent: DittoSpyder
150Disallow: /
151
152User-agent: SpankBot
153Disallow: /
154
155User-agent: BotALot
156Disallow: /
157
158User-agent: lwp-trivial/1.34
159Disallow: /
160
161User-agent: lwp-trivial
162Disallow: /
163
164User-agent: Wget/1.6
165Disallow: /
166
167User-agent: BunnySlippers
168Disallow: /
169
170User-agent: URLy Warning
171Disallow: /
172
173User-agent: Wget/1.5.3
174Disallow: /
175
176User-agent: LinkWalker
177Disallow: /
178
179User-agent: cosmos
180Disallow: /
181
182User-agent: moget
183Disallow: /
184
185User-agent: hloader
186Disallow: /
187
188User-agent: humanlinks
189Disallow: /
190
191User-agent: LinkextractorPro
192Disallow: /
193
194User-agent: Offline Explorer
195Disallow: /
196
197User-agent: Mata Hari
198Disallow: /
199
200User-agent: LexiBot
201Disallow: /
202
203User-agent: Web Image Collector
204Disallow: /
205
206User-agent: The Intraformant
207Disallow: /
208
209User-agent: True_Robot/1.0
210Disallow: /
211
212User-agent: True_Robot
213Disallow: /
214
215User-agent: BlowFish/1.0
216Disallow: /
217
218User-agent: JennyBot
219Disallow: /
220
221User-agent: MIIxpc/4.2
222Disallow: /
223
224User-agent: BuiltBotTough
225Disallow: /
226
227User-agent: ProPowerBot/2.14
228Disallow: /
229
230User-agent: BackDoorBot/1.0
231Disallow: /
232
233User-agent: toCrawl/UrlDispatcher
234Disallow: /
235
236User-agent: WebEnhancer
237Disallow: /
238
239User-agent: TightTwatBot
240Disallow: /
241
242User-agent: suzuran
243Disallow: /
244
245User-agent: VCI
246Disallow: /
247
248User-agent: Szukacz/1.4
249Disallow: /
250
251User-agent: QueryN Metasearch
252Disallow: /
253
254User-agent: Openfind
255Disallow: /
256
257User-agent: Xenu
258Disallow: /
259
260User-agent: Zeus
261Disallow: /
262
263User-agent: RepoMonkey Bait & Tackle/v1.01
264Disallow: /
265
266User-agent: RepoMonkey
267Disallow: /
268
269User-agent: Webster Pro
270Disallow: /
271
272User-agent: EroCrawler
273Disallow: /
274
275User-agent: LinkScan/8.1a Unix
276Disallow: /
277
278User-agent: Kenjin Spider
279Disallow: /