NeuralCrawl

Toyota / robots.txt snapshot

← back to toyota.com · fetched 2026-10-13T23:51:59Z (24d ago) · HTTP 200 · 2034 bytes · sha256 fd9533d9aa4b8623 · raw

final URL: https://www.toyota.com/robots.txt

1# ==================================================================
2
3# robots.txt — toyota.com
4
5# 2026-10-12
6
7# =======================================================================
8
9
10
11# GROUP 1 — Named AI Training Crawlers, including PerplexityBot — blocked everywhere
12
13
14
15User-agent: GPTBot
16
17User-agent: ClaudeBot
18
19User-agent: Google-Extended
20
21User-agent: Applebot-Extended
22
23User-agent: Meta-ExternalAgent
24
25User-agent: meta-externalagent
26
27User-agent: Bytespider
28
29User-agent: CCBot
30
31User-agent: cohere-ai
32
33User-agent: Diffbot
34
35User-agent: Omgilibot
36
37User-agent: Amazonbot
38
39User-agent: DuckAssistBot
40
41User-agent: anthropic-ai
42
43User-agent: PerplexityBot
44
45Disallow: /
46
47
48
49
50# GROUP 2 — Named Search and user-initiated agents — allowed on public pages, blocked on sensitive sections
51
52
53
54User-agent: ChatGPT-User
55
56User-agent: Claude-User
57
58User-agent: Perplexity-User
59
60User-agent: OAI-SearchBot
61
62User-agent: Claude-SearchBot
63
64User-agent: Applebot
65
66Disallow: /kbb*
67
68Disallow: /local-specials*
69
70Disallow: /what-fits-my-budget*
71
72Disallow: /dealers*
73
74Disallow: /payment-estimator*
75
76Disallow: /compare*
77
78Disallow: /configurator*
79
80Disallow: /search-inventory*
81
82Disallow: /request-a-quote*
83
84Disallow: /analytics/
85
86Disallow: /apps/recruiting/
87
88Disallow: /campaigns/
89
90Disallow: /resources/smartpath/
91
92Disallow: /espanol/resources/smartpath/
93
94Disallow: /resources/rct/
95
96Disallow: /espanol/resources/rct/
97
98
99
100# Catch-all — sensitive sections closed for everything else
101
102
103
104User-agent: *
105
106Disallow: /kbb*
107
108Disallow: /local-specials*
109
110Disallow: /what-fits-my-budget*
111
112Disallow: /payment-estimator*
113
114Disallow: /compare*
115
116Disallow: /configurator*
117
118Disallow: /search-inventory*
119
120Disallow: /request-a-quote*
121
122Disallow: /analytics/
123
124Disallow: /apps/recruiting/
125
126Disallow: /campaigns/
127
128Disallow: /resources/smartpath/
129
130Disallow: /espanol/resources/smartpath/
131
132Disallow: /resources/rct/
133
134Disallow: /espanol/resources/rct/
135
136
137
138
139
140Sitemap: https://www.toyota.com/sitemap_index.xml
141
142
143
144