NeuralCrawl

Toyota / robots.txt snapshot

← back to toyota.com · fetched 2026-10-01T01:22:45Z (1mo ago) · HTTP 200 · 1909 bytes · sha256 60e4242e3ed8b819 · raw

final URL: https://www.toyota.com/robots.txt

1# ==================================================================
2# robots.txt — toyota.com
3# 2026-09-15
4# =======================================================================
5
6# GROUP 1 — Named AI Training Crawlers, including PerplexityBot — blocked everywhere
7
8User-agent: GPTBot
9User-agent: ClaudeBot
10User-agent: Google-Extended
11User-agent: Applebot-Extended
12User-agent: Meta-ExternalAgent
13User-agent: meta-externalagent
14User-agent: Bytespider
15User-agent: CCBot
16User-agent: cohere-ai
17User-agent: Diffbot
18User-agent: Omgilibot
19User-agent: Amazonbot
20User-agent: DuckAssistBot
21User-agent: anthropic-ai
22User-agent: PerplexityBot
23Disallow: /
24
25
26# GROUP 2 — Named Search and user-initiated agents — allowed on public pages, blocked on sensitive sections
27
28User-agent: ChatGPT-User
29User-agent: Claude-User
30User-agent: Perplexity-User
31User-agent: OAI-SearchBot
32User-agent: Claude-SearchBot
33User-agent: Applebot
34Disallow: /kbb*
35Disallow: /local-specials*
36Disallow: /what-fits-my-budget*
37Disallow: /dealers*
38Disallow: /payment-estimator*
39Disallow: /compare*
40Disallow: /configurator*
41Disallow: /search-inventory*
42Disallow: /request-a-quote*
43Disallow: /analytics/
44Disallow: /apps/recruiting/
45Disallow: /campaigns/
46Disallow: /resources/smartpath/
47Disallow: /espanol/resources/smartpath/
48Disallow: /resources/rct/
49Disallow: /espanol/resources/rct/
50
51# Catch-all — sensitive sections closed for everything else
52
53User-agent: *
54Disallow: /kbb*
55Disallow: /local-specials*
56Disallow: /what-fits-my-budget*
57Disallow: /dealers*
58Disallow: /payment-estimator*
59Disallow: /compare*
60Disallow: /configurator*
61Disallow: /search-inventory*
62Disallow: /request-a-quote*
63Disallow: /analytics/
64Disallow: /apps/recruiting/
65Disallow: /campaigns/
66Disallow: /resources/smartpath/
67Disallow: /espanol/resources/smartpath/
68Disallow: /resources/rct/
69Disallow: /espanol/resources/rct/
70
71
72Sitemap: https://www.toyota.com/sitemap_index.xml
73