NeuralCrawl

Toyota / robots.txt

← back to toyota.com · change detected 24d ago (2026-10-13T23:51:59Z) · modified +140 −69

What changed

Diff old (2026-10-01T01:22:45Z) → new (2026-10-13T23:51:59Z)

@@ -1,73 +1,144 @@
1- # ==================================================================
2- # robots.txt — toyota.com
3- # 2026-09-15
4- # =======================================================================
1+ # ==================================================================
52 
6- # GROUP 1 — Named AI Training Crawlers, including PerplexityBot — blocked everywhere
3+ # robots.txt — toyota.com
4+
5+ # 2026-10-12
6+
7+ # =======================================================================
8+
79 
8- User-agent: GPTBot
9- User-agent: ClaudeBot
10- User-agent: Google-Extended
11- User-agent: Applebot-Extended
12- User-agent: Meta-ExternalAgent
13- User-agent: meta-externalagent
14- User-agent: Bytespider
15- User-agent: CCBot
16- User-agent: cohere-ai
17- User-agent: Diffbot
18- User-agent: Omgilibot
19- User-agent: Amazonbot
20- User-agent: DuckAssistBot
21- User-agent: anthropic-ai
22- User-agent: PerplexityBot
23- Disallow: /
10+
11+ # GROUP 1 — Named AI Training Crawlers, including PerplexityBot — blocked everywhere
12+
13+
14+
15+ User-agent: GPTBot
16+
17+ User-agent: ClaudeBot
18+
19+ User-agent: Google-Extended
20+
21+ User-agent: Applebot-Extended
22+
23+ User-agent: Meta-ExternalAgent
24+
25+ User-agent: meta-externalagent
26+
27+ User-agent: Bytespider
28+
29+ User-agent: CCBot
30+
31+ User-agent: cohere-ai
32+
33+ User-agent: Diffbot
34+
35+ User-agent: Omgilibot
36+
37+ User-agent: Amazonbot
38+
39+ User-agent: DuckAssistBot
40+
41+ User-agent: anthropic-ai
42+
43+ User-agent: PerplexityBot
44+
45+ Disallow: /
46+
47+
48+
49+
50+ # GROUP 2 — Named Search and user-initiated agents — allowed on public pages, blocked on sensitive sections
51+
52+
53+
54+ User-agent: ChatGPT-User
55+
56+ User-agent: Claude-User
57+
58+ User-agent: Perplexity-User
59+
60+ User-agent: OAI-SearchBot
61+
62+ User-agent: Claude-SearchBot
63+
64+ User-agent: Applebot
65+
66+ Disallow: /kbb*
67+
68+ Disallow: /local-specials*
69+
70+ Disallow: /what-fits-my-budget*
71+
72+ Disallow: /dealers*
73+
74+ Disallow: /payment-estimator*
75+
76+ Disallow: /compare*
77+
78+ Disallow: /configurator*
79+
80+ Disallow: /search-inventory*
81+
82+ Disallow: /request-a-quote*
83+
84+ Disallow: /analytics/
85+
86+ Disallow: /apps/recruiting/
87+
88+ Disallow: /campaigns/
89+
90+ Disallow: /resources/smartpath/
91+
92+ Disallow: /espanol/resources/smartpath/
93+
94+ Disallow: /resources/rct/
95+
96+ Disallow: /espanol/resources/rct/
97+
98+
99+
100+ # Catch-all — sensitive sections closed for everything else
101+
102+
103+
104+ User-agent: *
105+
106+ Disallow: /kbb*
107+
108+ Disallow: /local-specials*
109+
110+ Disallow: /what-fits-my-budget*
111+
112+ Disallow: /payment-estimator*
113+
114+ Disallow: /compare*
115+
116+ Disallow: /configurator*
117+
118+ Disallow: /search-inventory*
119+
120+ Disallow: /request-a-quote*
121+
122+ Disallow: /analytics/
123+
124+ Disallow: /apps/recruiting/
125+
126+ Disallow: /campaigns/
127+
128+ Disallow: /resources/smartpath/
129+
130+ Disallow: /espanol/resources/smartpath/
131+
132+ Disallow: /resources/rct/
133+
134+ Disallow: /espanol/resources/rct/
135+
136+
137+
138+
139+
140+ Sitemap: https://www.toyota.com/sitemap_index.xml
141+
24142 
143+
25144 
26- # GROUP 2 — Named Search and user-initiated agents — allowed on public pages, blocked on sensitive sections
27-
28- User-agent: ChatGPT-User
29- User-agent: Claude-User
30- User-agent: Perplexity-User
31- User-agent: OAI-SearchBot
32- User-agent: Claude-SearchBot
33- User-agent: Applebot
34- Disallow: /kbb*
35- Disallow: /local-specials*
36- Disallow: /what-fits-my-budget*
37- Disallow: /dealers*
38- Disallow: /payment-estimator*
39- Disallow: /compare*
40- Disallow: /configurator*
41- Disallow: /search-inventory*
42- Disallow: /request-a-quote*
43- Disallow: /analytics/
44- Disallow: /apps/recruiting/
45- Disallow: /campaigns/
46- Disallow: /resources/smartpath/
47- Disallow: /espanol/resources/smartpath/
48- Disallow: /resources/rct/
49- Disallow: /espanol/resources/rct/
50-
51- # Catch-all — sensitive sections closed for everything else
52-
53- User-agent: *
54- Disallow: /kbb*
55- Disallow: /local-specials*
56- Disallow: /what-fits-my-budget*
57- Disallow: /dealers*
58- Disallow: /payment-estimator*
59- Disallow: /compare*
60- Disallow: /configurator*
61- Disallow: /search-inventory*
62- Disallow: /request-a-quote*
63- Disallow: /analytics/
64- Disallow: /apps/recruiting/
65- Disallow: /campaigns/
66- Disallow: /resources/smartpath/
67- Disallow: /espanol/resources/smartpath/
68- Disallow: /resources/rct/
69- Disallow: /espanol/resources/rct/
70-
71-
72- Sitemap: https://www.toyota.com/sitemap_index.xml
73-